Parallel Data Laboratory

Backward Error Recovery in Redundant Disk Arrays (CMU-CS-94-193)

William V. Courtright II et al. — Thu, 13 May 2010 08:30:26 PDT

Redundant disk arrays are single fault tolerant, incorporating a layer of error handling not found in nonredundant disk systems. Recovery from these errors is complex, due in part to the large number of erroneous states the system may reach. The established approach to error recovery in disk systems is to transition directly from an erroneous state to completion. This technique, known as forward error recovery, relies upon the context in which an error occurs to determine the steps required to reach completion, which implies forward error recovery is design specific. Forward error recovery requires the enumeration of all erroneous states the system may reach and the construction of a forward path from each erroneous state. We propose a method of error recovery which does not rely upon the enumeration of erroneous states or the context in which errors occur. When an error is encountered, we advocate mechanized recovery to an error-free state from which an operation may be retried. Using a form of backward error recovery, we are able to manage the complexity of error recovery in redundant disk arrays without sacrificing performance.

A Redundant Disk Array Architecture for Efficient Small Writes (CMU-CS-94-170)

Daniel Stodolsky et al. — Thu, 13 May 2010 08:30:25 PDT

Parity encoded redundant disk arrays provide highly reliable, cost effective secondary storage with high performance for reads and large writes. Their performance on small writes, however, is much worse than mirrored disks - the traditional, highly reliable, but expensive organization for second ary storage. Unfortunately, small writes are a substantial portion of the I/O workload of many impor tant, demanding applications such as on-line transaction processing. This paper presents parity logging, a novel solution to the small write problem for redundant disk arrays. Parity logging applies journalling techniques to substantially reduce the cost of small writes. We provide detailed models of parity logging and competing schemes - mirroring, floating storage, and RAID level 5 - and verify these models by simulation. Parity logging provides performance competitive with mirroring, but with capacity overhead close to the minimum offered by RAID level 5. Finally, parity logging can exploit data caching more effectively than all three alternative approaches.

On-Line Data Reconstruction in Redundant Disk Arrays (CMU-CS-94-164)

Mark Holland — Thu, 13 May 2010 08:30:23 PDT

There exists a wide variety of applications in which data availability must be continuous, that is, where the system is never taken off-line and any interruption in the accessibility of stored data causes significant disruption in the service provided by the application. Examples include on-line transaction processing systems such as airline reservation systems, and automated teller networks in banking systems. In addition, there exist many applications for which a high degree of data availability is important, but continuous operation is not required. An example is a research and development environment, where access to a centrally-stored CAD system is often necessary to make progress on a design project. These applications and many others mandate both high performance and high availability from their storage subsystems.

Parity-based redundant disk arrays are very attractive storage alternatives for these systems because they offer both low cost per megabyte and high data reliability. Unfortunately such systems exhibit poor availability characteristics; their performance is severely degraded in the presence of a disk failure. This dissertation addresses the design of parity-based redundant disk arrays that offer dramatically higher levels of performance in the presence of failure than systems comprising the current state of the art.

We consider two primary aspects of the failure-recovery problem: the organization of the data and redundancy in the array, and the algorithm used to recover the lost data. We apply results from combinatorial theory to generate data and parity organizations that minimize performance degradation during failure recovery by evenly distributing all failure-induced workload over a larger-than-minimal collection of disks. We develop a reconstruction algorithm that is able to absorb for failure-recovery essentially all of the array's bandwidth that is not absorbed by the application process(es). Additionally, we develop a design for a redundant disk array targeted at extremely high availability through extremely fast failure recovery. This development also demonstrates the generality of the presented techniques.

A Redundant Disk Array Architecture for Efficient Small Writes (CMU-CS-93-200)

Daniel Stodolsky et al. — Thu, 13 May 2010 08:30:22 PDT

Parity encoded redundant disk arrays provide highly reliable, cost effective secondary storage with high performance for reads and large writes. Their performance on small writes, however, is much worse than mirrored disks - the traditional, highly reliable, but expensive organization for secondary storage. Unfortunately, small writes are a substantial portion of the I/O workload of many important, demanding applications such as on-line transaction processing. This paper presents parity logging, a novel solution to the small write problem for redundant disk arrays. Parity logging applies journalling techniques to substantially reduce the cost of small writes. We provide a detailed analysis of parity logging and competing schemes - mirroring, floating storage, and RAID level 5 - and verify these models by simulation. Parity logging provides performance competitive with mirroring, the best of the alternative single failure tolerating disk array organizations. However, its overhead is close to the minimum offered by RAID level 5. Finally, parity logging can exploit data caching much more effectively than all three alternative approaches.

A Status Report on Research in Transparent Informed Prefetching (CMU-CS-93-113)

R. Hugo Patterson et al. — Thu, 13 May 2010 08:30:21 PDT

This paper focuses on extending the power of caching and prefetching to reduce file read latencies by exploiting application level hints about future I/O accesses. We argue that systems that disclose high-level knowledge can transfer optimization information across module boundaries in a manner consistent with sound software engineering principles. Such Transparent Informed Prefetching (TIP) systems provide a technique for converting the high through put of new technologies such as disk arrays and log-structured file systems into low latency for applications. Our preliminary experiments show that even without a high-throughput I/O sub system TIP yields reduced execution time of up to 30% for applications obtaining data from a remote file server and up to 13% for applications obtaining data from a single local disk. These experiments indicate that greater performance benefits will be available when TIP is integrated with low level resource management policies and highly parallel I/O subsystems such as disk arrays.

A Trace-Driven Comparison of Algorithms for Parallel Prefetching and Caching (CMU-CS-96-174)

Tracy Kimbrel et al. — Wed, 12 May 2010 08:51:06 PDT

High-performance I/O systems depend on prefetching and caching in order to deliver good performance to applications. These two techniques have generally been considered in isolation, even though there are significant interactions between them; a block prefetched too early reduces the effectiveness of the cache, while a block cached too long reduces the effectiveness of prefetching. In this paper we study the effects of several combined prefetching and caching strategies for systems with multiple disks. Using disk-accurate trace-driven simulation, we explore the performance characteristics of each of the algorithms in cases in which applications provide full advance knowledge of accesses using hints. Some of the strategies have been published with theoretical performance bounds, and some are components of systems that have been built. One is a new algorithm that combines the desirable characteristics of the others. We find that when performance is limited by I/O stalls, aggressive prefetching helps to alleviate the problem; that more conservative prefetching is appropriate when significant I/O stalls are not present; and that a single, simple strategy is capable of doing both.

Understanding Customer Dissatisfaction with Underutilized Distributed File Servers (CMU-CS-96-158)

Erik Riedel et al. — Wed, 12 May 2010 08:51:05 PDT

An important trend in the design of storage subsystems is a move toward direct network attachment. Network-attached storage offers the opportunity to off-load distributed file system functionality from dedicated file server machines and execute many requests directly at the storage devices. For this strategy to lead to better performance as perceived by users, the response time of distributed operations must improve. In this paper, we analyze measurements of an Andrew File System (AFS) server that we recently upgraded in an effort to improve client performance in our laboratory. While the original server's overall utilization was only about 3%, we show how burst loads were sufficiently intense to lead to periods of poor response time significant enough to trigger customer dissatisfaction. In particular, we show how, after adjusting for network load and traffic to non-project servers, 50% of the variation in client response time was explained by variation in server CPU utilization. That is, clients saw long response times in large part because the server was often over-utilized when it was used at all. Using these measures, we see that off-loading file server work in a network-attached storage architecture has the potential to benefit user response time. Computational power in such a system scales directly with storage capacity, so the slowdown during burst periods should be reduced.

A Case for Network-Attached Secure Disks (CMU-CS-96-142)

Garth A. Gibson et al. — Wed, 12 May 2010 08:51:03 PDT

By providing direct data transfer between storage and client, network-attached storage devices have the potential to improve scalability (by removing the server as a bottleneck) and performance (through network striping and shorter data paths). Realizing the technology's full potential requires careful consideration across a wide range of file system, networking and security issues. To address these issues, this paper presents two new network-attached storage architectures. (1) Networked SCSI disks (NetSCSI) are network-attached storage devices with minimal changes from the familiar SCSI interface (2) Network-attached secure disks (NASD) are drives that support independent client access to drive provided object services. For both architectures, we present a sketch of repartitionings of distributed file system functionality, including a security framework whose strongest levels use tamper resistant processing in the disks to provide action authorization and data privacy even when the drive is in a physically insecure location.

Using AFS and NFS, trace results suggest that NetSCSI can reduce file server load during a burst of AFS activity by a factor of about 2; for the NASD architecture, server load (during burst activity) can be reduced by a factor of about 4 for AFS and 10 for NFS.

A Structured Approach to Redundant Disk Array Implementation (CMU-CS-96-137)

William V. Courtright II et al. — Wed, 12 May 2010 08:51:02 PDT

Error recovery in redundant disk arrays is typically performed in an ad hoc fashion, requiring architecture-specific code which limits extensibility and is difficult to verify. In this paper, we describe a technique for automating the execution of redundant disk array operations, including recovery from errors, independent of array architecture. Our approach employs a graphical representation of array operations and a two-phase error-recovery scheme we refer to as roll-away error recovery. We demonstrate the validity of this approach in RAIDframe, a prototyping framework that separates architectural policy from execution mechanism. RAIDframe facilitates rapid proto- typing of new RAID architectures by localizing modifications. In addition, RAIDframe-implemented architectures run the same code when configured as an event-driven simulator, a user-level application managing raw disks, and as a Digital Unix device-driver capable of mounting a file system. Evaluation shows that RAIDframe performance is equivalent to less complex array implementations and that case studies of RAID levels 0, 1, 4, 5, 6, and parity declustering achieve expected performance

RAIDframe: Rapid Prototyping for Disk Arrays (CMU-CS-95-200)

William V. Courtright II et al. — Wed, 12 May 2010 08:51:00 PDT

In recent years, researchers have introduced a multitude of redundant disk array architectures. Unfortunately, using the tradi-tional manual firmware-design approach employed by storage sys-tem designers, implementing control software for these redundant disk array architectures has led to long product-development times, yielded uncertain product reliability, and inhibited designers from exploring the benefits of new architectures. What is needed is array prototyping technology that makes it easier to experiment with design changes while also making it easier to ensure correct functioning of these changes.

We introduce RAIDframe, a framework for rapid prototyping and evaluation of redundant disk arrays. Using a graphical pro-gramming abstraction and a mechanized execution strategy, we are able to quickly construct working prototypes which can immedi-ately be evaluated each of three environments: a device driver run-ning against real disks, a user process running against real disks, or an event-driven simulator. This paper describes the basic structure of RAIDframe as well as our experiences with it.

The Scotch Parallel Storage Systems (CMU-CS-95-107)

Garth A. Gibson et al. — Wed, 12 May 2010 08:50:58 PDT

To meet the bandwidth needs of modern computer systems, parallel storage systems are evolving beyond RAID levels 1 through 5. The Parallel Data Lab at Carnegie Mellon University has constructed three Scotch parallel storage testbeds to explore and evaluate five directions in RAID evolution: first, the development of new RAID architectures to reduce the cost/performance penalty of maintaining redundant data; second, an extensible software framework for rapid prototyping of new architectures; third, mechanisms to reduce the complexity of and automate error-handling in RAID subsystems; fourth, a file system extension that allows serial programs to exploit parallel storage; and lastly, a parallel file system that extends the RAID advantages to distributed, parallel computing environments. This paper describes these five RAID evolutions and the testbeds in which they are being implemented and evaluated.

Informed Prefetching and Caching (CMU-CS-97-204)

Russel Hugo Patterson — Tue, 11 May 2010 08:11:42 PDT

Disk arrays provide the raw storage throughput needed to balance rapidly increasing processor performance. Unfortunately, many important, I/O-intensive applications have serial I/O workloads that do not benefit from array parallelism. The performance of a single disk remains a bottleneck on overall performance for these applications. In this dissertation, I present aggressive, proactive mechanisms that tailor file-system resource management to the needs of I/O-intensive applications. In particular, I will show how to use application-disclosed access patterns (hints) to expose and exploit I/O parallelism, and to dynamically allocate file buffers among three competing demands: prefetching hinted blocks, caching hinted blocks for reuse, and caching recently used data for unhinted accesses. My approach estimates the impact of alternative buffer allocations on application elapsed time and applies run-time cost-benefit analysis to allocate buffers where they will have the greatest impact. I implemented TIP, an informed prefetching and caching manager, in the Digital UNIX operating system and measured its performance on a 175 MHz Digital Alpha workstation equipped with up to 10 disks running a range of applications. Informed prefetching on a ten-disk array reduces the wall-clock elapsed time of computational physics, text search, scientific visualization, relational database queries, speech recognition, and object linking by 10-84% with an average of 63%. On a single disk, where storage parallelism is unavailable and avoiding disk accesses is most beneficial, informed caching reduces the elapsed time of these same applications by up to 36% with an average of 13% compared to informed prefetching alone. Moreover, applied to multiprogrammed, I/O-intensive workloads, TIP increases overall throughput.

Active Disks: Remote Execution for Network-Attached Storage (CMU-CS-97-198)

Erik Riedel et al. — Tue, 11 May 2010 08:11:40 PDT

The principal trend in the design of computer systems is the expectation of much greater computational power in future generations of microprocessors. This trend applies to embedded systems as well as host processors. As a result, devices such as storage controllers have excess capacity and growing computational capabilities. Storage system designers are exploiting this trend with higher-level interfaces to storage and increased intelligence inside storage devices. One development in this direction is Network-Attached Secure Disks (NASD) which attaches storage devices directly to the network and raises the storage interface above the simple (fixed-size block) memory abstraction of SCSI. This allows devices more freedom to provide efficient operations; promises more scalable subsystems by offloading file system and storage management functionality from dedicated servers; and reduces latency by executing common case requests directly at storage devices. In this paper, we push this increasing computation trend one step further. We argue that application-specific code can be executed at storage devices to make more effective use of device, host and interconnect resources and significantly improve application I/O performance. Remote execution of code directly at storage devices allows filter operations to be performed close to the data; enables support of timing-sensitive transfers and application-aware scheduling of access and transfer; allows management functions to be customized without requiring firmware changes; and makes possible more complex or specialized operations than a general-purpose storage interface would normally support.

Security for Network Attached Storage Devices (CMU-CS-97-185)

Howard Gobioff et al. — Tue, 11 May 2010 08:11:39 PDT

This paper presents a novel cryptographic capability system addressing the security and performance needs of network attached storage systems in which file management functions occur at a different location than the file storage device. In our NASD system file managers issue capabilities to client machines, which can then directly access files stored on the network attached storage device without intervention by a file server. These capabilities may be reused by the client, so that interaction with the file manager is kept to a minimum. Our system emphasizes performance and scalability while separating the roles of decision maker (issuing capabilities) and verifier (validating a capability). We have demonstrated our system with adaptations of both the NFS and AFS distributed file systems using a prototype NASD implementation.

Practical and Theoretical Issues in Prefetching and Caching (CMU-CS-97-181)

Andrew Tomkins — Tue, 11 May 2010 08:11:38 PDT

This thesis has two parts, the first more practical, and the second more theoretical. The first part considers informed prefetching and caching in which an application provides information about its upcoming I/O accesses to the operating system, allowing the system to prefetch data and to make informed cache replacement decisions. I compare existing algorithms for this problem using trace-driven simulation, and use the results to develop a new algorithm that performs better than previous approaches, again under trace-driven simulation.

The second part considers weighted caching, a theoretical problem from the domain of on-line algorithms. I present an algorithm with competitive ratio O(log2 k) on (k + 1)-point spaces, the first poly-logarithmic ratio for this problem. I also give an almost-tight lower bound of W(log k) for any weighted caching problem on at least k + 1 points. I then show a connection between this problem and a new on-line k-server model in which the servers may rearrange themselves without cost during "free-time" between requests, and describe a series of results in the free-time model.

RAIDframe: A Rapid Prototyping Tool for RAID Systems (CMU-CS-97-142)

William V. Courtright II et al. — Tue, 11 May 2010 08:11:37 PDT

Redundant disk arrays provide highly-available, high-performance disk storage to a wide variety of applications. Because these applications often have distinct cost, performance, capacity, and availability requirements, researchers continue to develop new array architectures. RAIDframe was developed to assist researchers in the implementation and evaluation of these new architectures. It was designed specifically to reduce the burden of implementation by restricting code changes to mapping, algorithms and other functions that are known to be specific to an array architecture. Algorithms are executed using a general mechanism which automates the recovery from device errors, such as a failed disk read. RAIDframe enables a single implementation to be evaluated in a self-contained simulator, or against real disks as either a user process or a functional device driver.

A Transactional Approach to Redundant Disk Array Implementation (CMU-CS-97-141)

William V. Courtright II — Tue, 11 May 2010 08:11:35 PDT

Redundant disk arrays are a popular method of improving the dependability and performance of disk storage and an ever increasing number of array architectures are being proposed to balance cost, performance, and dependability. Despite their differences, there is a great deal of commonality between these architectures; unfortunately, it appears that current implementations are not able to effectively exploit this commonality due to their ad hoc approach to error recovery. Such techniques rely upon a case-by-case analysis of errors, a manual process that is tedious and prone to mistakes. For each distinct error scenario, a unique procedure is implemented to remove the effects of the error and complete the affected operation. Unfortunately, this form of recovery is not easily extended because the analysis must be repeated as new array operations and architectures are introduced. Transaction-processing systems utilize logging techniques to mechanize the process of recovering from errors. However, the expense of guaranteeing that all operations can be undone from any point in their execution is too expensive to satisfy the performance and resource requirements of redundant disk arrays.

This dissertation describes a novel programming abstraction and execution mechanism based upon transactions that simplifies implementation. Disk array algorithms are modeled as directed acyclic graphs: the nodes are actions such as "XOR" and the arcs represent data and control dependencies between them. Using this abstraction, we implemented eight array architectures in RAIDframe, a framework for prototyping disk arrays. Code reuse was consistently above 90%. The additional layers of abstraction did not affect the response time and throughput characteristics of RAIDframe; however, RAIDframe consumes 60% more CPU cycles than a hand-crafted non-redundant implementation. RAIDframe employs roll-away error recovery, a novel scheme for mechanizing the execution of disk array algorithms without requiring that all actions be undoable. A barrier is inserted into each algorithm: failures prior to the barrier result in rollback, relying upon undo information. Once the barrier is crossed, the algorithm rolls forward to completion, and undo records are unnecessary. Experiments revealed this approach to have identical performance to that of non-logging schemes.

Filesystems for Network-Attached Secure Disks (CMU-CS-97-118)

Garth A. Gibson et al. — Tue, 11 May 2010 08:11:33 PDT

Network-attached storage enables network-striped data transfers directly between client and storage to pro vide clients with scalable bandwidth on large transfers. Network-attached storage also decouples policy and enforcement of access control, avoiding unnecessary reverification of protection checks, reducing file manager work and increasing scalability. It eliminates the expense of a server computer devoted to copying data between peripheral network and client network. This architecture better matches storage technology's sus tained data rates, now 80 Mb/s and growing at 40% per year. Finally, it enables self-managing storage to counter the increasing cost of data management. The availability of cost-effective network-attached storage depends on it becoming a storage commodity, which in turn depends on its utility to a broad segment of the storage market. Specifically, multiple distributed and parallel filesystems must benefit from network-attached storage's requirement for secure, direct access between client and storage, for reusable, asynchronous access protection checks, and for increased license to efficiently manage underlying storage media. In this paper, we describe a prototype network-attached secure disk interface and filesystems adapted to network-attached stor age implementing Sun's NFS, Transarc's AFS, a network-striped NFS variant, and an informed prefetching NFS variant. Our experimental implementations demonstrate bandwidth and workload scaling and aggressive optimization of application access patterns. Our experience with applications and filesystems adapted to run on network-attached secure disks emphasizes the much greater cost of client network messaging relative to peripheral bus messaging, which offsets some of the expected scaling results.

Integrity and Performance in Network Attached Storage (CMU-CS-98-182)

Howard Gobioff et al. — Tue, 11 May 2010 07:48:51 PDT

Computer security is of growing importance in the increasingly networked computing environment.This work examines the issue of high-performance network security, specifically integrity, by focusing on integrating security into network storage system. Emphasizing the cost-constrained environment of storage, we examine how current software-based cryptography cannot support storage's Gigabit/sec transfer rates. To solve this problem, we introduce a novel message authentication code, based on stored message digests. This allows storage to deliver high-performance, a factor of five improvement in our prototype's integrity protected bandwidth, without hardware acceleration for common read operations. For receivers, where precomputation cannot be done, we outline an inline message authentication code that minimizes buffering requirements.

Selected Reports: Fall 1997 Software Systems Course (CMU-CS-98-103)

Garth Gibson et al. — Tue, 11 May 2010 07:48:49 PDT

This technical report contains seven final project reports contributed by sixteen participants in CMU's Systems Software introductory graduate course offered by Professor Garth Gibson. This course studies the design and analysis of operating systems and distributed systems through a series of background lectures, paper readings, guest lectures and group projects. Projects were done in groups of two or three, required some kind of implementation and evaluation pertaining to the classroom material, but with the topic of these projects left up to each group. Final reports were held to the standard of a systems conference paper submission; a standard well met by the majority of the completed projects, albeit with less thoroughness in the related work category than is expected in most conferences. The reports that follow cover a broad range of topics. Specifically, these reports describe implementations and experimentation with: secure file systems when servers and administrators are untrusted; proportional share allocation for processor scheduling and its interaction with kernel realities such as locks; eventually serializable replicated databases with constant-order dependency checking; compressed file data structures optimized to specific access patterns; atomic, shared object semantics for distributed computing in JAVA; transaction semantics for federated agent databases; and user-level file service offering enhanced memory caching for remote files. All reports include implementations and experimentation. Two involve operating system kernel changes, four use a middleware /library approach and one implements a client/server system. Three involve Linux specific modifications, one is specific to FreeBSD, one extends JAVA programming, and one exploits MPI communications. Evaluations include microbenchmark measurements, formal correctness evaluation, synthetic benchmarks, and more than a couple specifically developed application codes. While not all of these reports report definitely and positively, all involve novelty in either the systems explored or the applications applied and all are worth reading.