ie ai i \\enn | ni nm a oe

ti rll

{jj

ee TT TT

CONFERENCE

PROCEEDINGS

Systems Administration (LISA VI)

| October 19-October 23, 1992 Long Beach, California

SONIGHIOONd (IA VSIT) JONAWAANOD NOLLVALLSININGYV SINA.LSAS

For additional copies of these proceedings contact

USENIX Association 2560 Ninth Street, Suite 215 Berkeley, CA 94710 USA Telephone: 510-528-8649

The price is $23 for members and $30 for nonmembers.

Outside the U.S.A. and Canada, please add $12 per copy for postage (via air printed matter).

Past USENIX Large Installation Systems Administration Workshop and Conference Proceedings (price: member/nonmember)

Large Installation Systems Admin. I Workshop 1987 Phildelphia, PA $4/$4 Large Installation Systems Admin. II Workshop 1988 Monterey, CA $8/S8 Large Installation Systems Admin. III Workshop 1989 ~—- Austin, TX $13/$13 Large Installation Systems Admin. IV Conference 1990 Colorado Spgs,CO $15/$18 Large Installation Systems Admin. V Conference 1991 San Diego, CA $20/S23

Outside the U.S.A. and Canada, please add $8 per copy for postage (via air printed matter).

Copyright 1992 by The USENIX Association. ISBN 1-880446-47-2

All rights reserved. This volume is published as a collective work. Rights to individual papers remain with the author or the author’s employer.

AFS is a trademark of Transarc, Inc. Budtool is probably a trademark of Delta Microsystems, Inc. Ethernet is a trademark of Xerox Corp. FastPath is a trademark of Shiva Corp.

HP9000s720 is a trademark of Hewlett Packard Corp. INGRES, INGRES/QUEL, and INGRES/EQUEL are trademarks of Ingres Corporation. Kerberos, Zephyr, Moira, Hesiod, and Athena are trademarks of MIT. Legato NetWorker is a trademark of Legato Systems, Inc. MS-DOS is a trademark of Microsoft Corp.

Macintosh is a trademark of Apple Computer, Inc. MultiNet is a trademark of TGV, Inc.

Multimax is a trademark of Encore Computer Corporation.

NIS, NFS, Sun-4, SPARCsystem, SPARCstation, SPARCserver, and SunOS are trademarks of Sun Microsystems, Inc. NeXt, NeXTstep, and NeXTcube are registered trademarks of NeXT Computer, Inc. Novell and NetWare are trademarks of Novell, Inc.

PC/TCP, FTP Software are trademarks of FTP Software, Inc.

PDP-11, DECSYSTEM-20, Ultrix, DEC, VAX, “VAX, and VMS are trademarks of Digital Equipment Corp. RS6000m560, AIX, IBM, OS/2, PC-DOS, DOS, VM, CMS are trademarks of IBM Corp. Six Sigma is a trademark of Motorola StarSENTRY is a trademark of NCR Corporation.

SUMMUS is a trademark of SUMMUS Corporation.

UNIX and System V are registered trademarks of Unix Systems Laboratories. Yellow Pages is a trademark of British Telecom, Inc. cisco is a trademark of cisco Systems, Inc.

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and USENIX was aware of a trademark claim, the designations have been printed in caps or initial caps.

Printed in the United States of America on 50% recycled paper, 10-15% post consumer waste. &

USENIX Association

Proceedings of the Sixth Systems Administration Conference (LISA VI)

October 19-23, 1992 Long Beach, CA, USA

TABLE OF CONTENTS

ACkNOWIEUBMERES sscssseacasccssessczenscssceessenasscassvxsaszestaveeanecassncestnsessieszaahoanem cance eseannsstixanvensaeaesaueseaeeT IT RRET Vv PRCLSCE ..caccaereescteates Sec cestes Soigsis5 gach ten stp ta tUeeyQeGe UNG g6a 50 Se daw ab ices Sab v ARSE SAENE Gas USAR RARE vi TAMEDOR INCOR eecpedasswnaeinc vreswestercesthonncaceaicescenaaviusias chuvce uencsensiceneneUussieseekatsawsaedontesanncsuibpeay desk snnaavedtaninaSodend vii

Plenary Session Wednesday (9:00-10:00) Chair: Trent Hein

Opening Remarks and Announcements Trent Hein, XOR Network Engineering

Keynote Address: A Retrospective on Downsizing Doug Kingston, Morgan Stanley and Company, Inc.

NFS and Workstation Performance Wednesday (10:30-11:30) Chair: Rob Kolstad

Effective Use of Local Workstation Disks in an NFS Network .....ccccccssscssssscsseesscsseeeseesecseesecescseaesensens 1 Paul Anderson, University of Edinburgh

Optimal Routing of IP Packets to Multi-Homed Servers .o..ccsessssssssssssssescesssesesesessssseacacseesesseaeseseeeees 9 Karl L. Swartz, Stanford Linear Accelerator Center

NFS and Workstation Performance, II

Wednesday (1:00-2:00) Chair: Rik Farrow

LADDIS: A Multi-Vendor and Vendor-Neutral SPEC NFS Benchmark .......ccccscsssssscscsesesessesssssscsesees 17 Andy Watson & Bruce Nelson, LADDIS Group & Auspex Systems

NFS Performance And Network Loading ..ssccccsssesssssescscsescsescecsesescsesescscsescaeecececscscasscsvavsvscasanscseeesaes 33

Hal L. Stern & Brian L. Wong, Sun Microsystems Computer Corporation

UNIX as the All-Purpose Computing Environment

Wednesday (2:30-4:00) Chair: Pat Parseghian Dropping the Mainframe Without Crushing the Users: Mainframe to Distributed UNIX in Nine UR ERAN EEG ween Zac conan cea seul Sec gpg te ome otsians iNet 39

Peter Van Epp & Bill Baines, Simon Fraser University

Is Centralized System Administration the ANSWEr? ....cccscsssscssesscsesessesesessscsesssscsssesessssesssssssscesavsceseases 55 Peg Schafer, BBN

Panel: Models of System Administration Pat Parseghian, AT&T Bell Laboratories

LISA VI - October 19-23, 1992 - Long Beach, CA i

System Administrator Training and Customer Satisfaction Wednesday (4:30-5:00) Chair: Jeff Polk

Customer Satisfaction Metrics and MEaSUFeMENt .....sssscsesessscseeesssssereceeeesssssesesaeesessrseecesesseeaneeeeeereeenes 63 Carol Kubicki, Motorola Cellular Infrastructure Group

Request: A Tool for Training New Sys Admins and Managing Old Ones ....sssesssesssreesrseesreneseereees 69 James M. Sharp, Lawrence Livermore National Laboratory

Mass Host Configuration and Duplication

Thursday (9:30-10:30) Chair: Trent Hein

Typecast: Beyond Cloned Hosts .......ssssssssssssssssssesssseseseenssenessnssnesenssnsesssssssssscssensseseessnsssscesensssesensenenees 73 Elizabeth D. Zwicky, SRI International

So Many Workstations, So Little Time ....cccccccssscsseseseseenerenesetenseetensseneneseenetsnenessssssssenseessesersnerereneees 79

Helen E. Harrison, SAS Institute Inc.

Mass Host Configuration and Duplication, II

Thursday (11:00-12:00) Chair: Rob Kolstad

Mkserv Workstation Customization and Privatization ....cccsssessessssessesecssesecsseeesesnecseesecaeenecneeeeeaeeaeens 89 Mark Rosenstein & Ezra Peisach, MIT Information Systems

AUTOLOAD: The Network Management System ..csssessesssesssseseeeeccsseessessesseneeetseseeseeeeseeaseesseaesaeseeeees 97 Dieter Pukatzki, Leading Edge Technology Transfer; Johann Schumann, Institut fiir Informatik

System Administration Potpourri, I Thursday (1:30-3:30) Chair: Herb Morreale

ipasswd Proactive Password Security ...cscscssssesssssererereesesesesenescseesseseeeneseseseneseasssseseseseeeseeseseasseeneees 105 Jarkko Hietaniemi, Helsinki University of Technology

DeeJay The Dump Jockey: A Heterogeneous Network Backup System Melissa Metz & Howie Kaye, Columbia University Academic Information Systems

Dealing with Lame Delegations ..........cssscsssssssersesesssosscscscsesenesenscesesesesesssescaeneasecsnacsesssessessessesorseseess 127 Bryan Beecher, University of Michigan

Majordomo: How I Manage 17 Mailing Lists Without Answering "-request"” Mail .......ssssesseseereeseeees 135 D. Brent Chapman, Great Circle Associates

ii LISA VI - October 19-23, 1992 Long Beach, CA

Distributed Software Management, I

Thursday (4:00-5:30) Chair: Jeff Polk

SysView: A User-friendly Environment for Administration of Distributed UNIX Systems _ ............+. 143 Philippe Coq & Sylvie Jean, Bull S.A. France

Depot: A Tool for Managing Software Environments .......s.scscsesssssssesssssesesesescsesescscscececsvscscseececenseeaneees 151

Wallace Colyer & Walter Wong, Carnegie Mellon University

Software Distribution and Management in a Networked Environment .....c.ccccsssesesssssssssessessescsesvsesecsens 163 Ram R. Vangala & Michael J. Cripps, NCR; Raj G. Varadarajan, AT&T

Distributed Configuration Management Friday (9:30-11:00) Chair: Pat Parseghian

“‘Nightly’’: How to Handle Multiple Scripts on Multiple Machines with One Configuration File ... 171 Jeff Okamoto, Hewlett-Packard

Overhauling Raist:forthe "QOS. sesccvessispcvasesavestessariveastsdecscanapdosecossavessnesacensdedeoacaovedeceases eancdensvsvovesdecavese 175 Michael A. Cooper, University of Southern California

doits A. Newvrork Software: Management Tol. srisssiscnvssncienmacieroonivasascivientaeswaviccueaninisnwnnasesnvogens 189 Mark Fletcher, SAS Institute Inc.

Monitoring System and User Problems Friday (11:30-12:30) Chair: Rik Farrow

PLUSH AL ROQuest Mann cement BYSUSIL., - siiconsnxeisiasnpissimesanadeceatasiassimmeccanunnioeareiseenonatcamue 197 David Koblas, Independent Consultant; Paul M. Moriarty, cisco Systems, Inc.

buzzerd: Automated Systems Monitoring with Notification in a Network Environment. .....s.c.ssseesee 203 Darren R. Hardy & Herb M. Morreale, XOR Network Engineering, Inc.

System Administration Potpourri, II Friday (2:00-3:30) Chair: Trent Hein

bbn-public Contributions from the User Community ......sccsssssessssssssesssesssssessessecsscssssvessesnsencsucssssucssen 211 Peg Schafer, BBN

user-setup: A System for Custom Configuration of User Environments, or Helping Users Help RERUN Seewearesitan etc nt cetahvacsn acca ssiehesabinionis sia eadean taiansosasioabnenh meeodaotuosereniek oapiataias 215 Richard Elling & Matthew Long, Auburn University

TelLand TR Tadis tor the:System Adminisinator: sacumancccunninniecmunnsnaaneionannadarns 225 Brad Morrison & Karl Lehenbauer, Paranet, Inc.

LISA VI - October 19-23, 1992 Long Beach, CA iii

Distributed Software Management, II Friday (4:00-5:00) Chair: Herb Morreale

Concurrent Network Management with a Distributed Management T00] ...ssesssessesereseseersrerssnsersreneeees 235 R. Lehman, G. Carpenter, & N. Hien, IBM T. J. Watson Research Center

nlp: A Network Printing To] ..ssssssssssesseeesessescseesessensersacenensseseesesrsssssssssnsesssanseseeessesassesescsseeneneserees 245 Mark Fletcher, SAS Institute Inc.

iv LISA VI - October 19-23, 1992 —- Long Beach, CA

ACKNOWLEDGMENTS

PROGRAM CHAIR Trent Hein, XOR Network Engineering, Inc.

PROGRAM COMMITTEE Rik Farrow, UNIX World

Jeff Forys, University of Utah

John Hardt, Martin Marietta Astronautics Rob Kolstad, Berkeley Software Design, Inc. Herb Morreale, XOR Network Engineering, Inc.

Pat Parseghian, AT&T Bell Laboratories Jeff Polk, Berkeley Software Design, Inc.

PROCEEDINGS PRODUCTION Rob Kolstad, Berkeley Software Design, Inc. Malloy Lithographing, Inc.

Carolyn Carr, USENIX Association

ALTERNATE TRACK COORDINATOR Steve Simmons, /ndustrial Technology Institute

BOF COORDINATOR Arch Mott, Protocol Engines, Inc.

TUTORIAL COORDINATOR Daniel V. Klein, USENIX Association

TERMINAL ROOM COORDINATOR Barb Dyker, University of Colorado, Boulder

VENDOR DISPLAY COORDINATOR John Donnelly, Seminars, Meetings, and Exhibits

USENIX MEETING PLANNER Judith F. Desharnais, USENIX Association

LISA VI - October 19-23, 1992 - Long Beach, CA

PREFACE

I have found LISA to always be the most amazing USENIX gathering. As a breed, system administrators seem to have a number of rather special traits. Among them, strong personalities and fastidiousness seem to always float to the top of the list. These are what have made LISA a raging success over the years, but are also what have forced LISA to be a carefully balanced technical and social gathering.

For the first time this year, LISA is a full-tracked 5 day conference, having grown from a small "workshop" in just 6 years. These proceedings are just a few pages longer than the most recent "general" USENIX conference (San Antonio). After looking at the quality of papers we have this year, I think I can safely say that it would be unfair for anyone to call LISA a "minor-league" conference from here on out. As system administrators, we can be all be proud of LISA, and I want to thank all of you reading this for making that happen.

Also, for the first time, the LISA (Large Installation System Administration) name is being kept for historical purposes only, and we are now welcoming system administrators from sites of all sizes, from one host to ten thousand. I feel that our outstanding technical program this year will address interests of this wide attendee base. Whether you’re an old or new LISA attendee, please make the effort to share your views and make your interests known in the on-the-fly discussions that so often pop up in the lobbies or bars of a USENIX conference. The more our community interconnects, the stronger we’ll be as a profession.

Organizing a conference like LISA requires an incredible amount of behind-the-scenes time and effort from an army of dedicated volunteers. Special thanks goes to all the members of my program committee, who have all been fantastic to work with. I would also like to thank our Rob Kolstad for acting as Board Liaison and Proceedings Typesetter, Arch Mott for volunteering as BOF Coordinator, Barb Dyker for organizing the terminal room, John Donnelly for planning the vendor displays, and Steve Simmons for organizing the alternate track. Last but most certainly not least, I would like to extend special thanks to Judy DesHarnais, who as our meeting planner has been infinitely patient and incredibly helpful to me as this year’s program chair.

Trent R. Hein Program Chair

vi LISA VI - October 19-23, 1992 - Long Beach, CA

Paul Anderson Bill Baines

Bryan Beecher

G. Carpenter

D. Brent Chapman Wallace Colyer Michael A. Cooper Philippe Coq Michael J. Cripps Richard Elling Peter Van Epp Mark Fletcher Mark Fletcher Darren R. Hardy Helen E. Harrison N. Hien

Jarkko Hietaniemi Sylvie Jean Howie Kaye David Koblas Carol Kubicki Karl Lehenbauer R. Lehman

LISA VI - October 19-23, 1992 - Long Beach, CA

AUTHOR INDEX

39 127 235 135 151 175 143 163 215

39 189 245 203

79 235 105 143 115 197

63 225 235

Matthew Long Melissa Metz Paul M. Moriarty Herb M. Morreale Brad Morrison Bruce Nelson

Jeff Okamoto Ezra Peisach Dieter Pukatzki Mark Rosenstein Peg Schafer

Peg Schafer Johann Schumann James M. Sharp Hal L. Stern

Karl L. Swartz Ram R. Vangala Raj G. Varadarajan Andy Watson Brian L. Wong Walter Wong Elizabeth D. Zwicky

215 115 197 203 225 17 171 89 97 89 25 211 97 69 33

163 163 17 oS 151 73

vii

Effective Use of Local Workstation Disks in an NFS Network

Paul Anderson University of Edinburgh

ABSTRACT

This paper presents an analysis of the NFS traffic generated by a typical group of Unix workstations. The relative advantages and disadvantages of holding different categories of data on local workstation disks are then discussed, in the light of these results. Finally, a program is described that automatically reconfigures a local disk, at regular intervals, to transparently hold copies of the most frequently used programs and data. Trace-driven simulations of this program, using the NFS trace data, and experience of the system in practice, show that considerable savings can be made in server and network traffic using

comparatively small amounts of local disk space.

Introduction

A typical Unix network includes one or more fileservers that provide the bulk of the filestore for the workstations on the network. The use of central servers is usually more cost-effective in terms of storage and easier to administer than a highly distri- buted filesystem. However, many individual works- tations also incorporate their own, much smaller disks, that are generally used to hold frequently accessed data, improving the workstation perfor- mance and reducing the load on the servers and the network.

Special filesystems, such as the Andrew File System [1] (AFS), are capable of utilising local disk Space as a filesystem cache that automatically attempts to minimize remote file accesses. How- ever, the most common remote filesystem for Unix workstations is probably Sun’s Network Filesys- tem [2] (NFS) which provides no such inbuilt facili- ties. In a "traditional" NFS network, a small local disk would typically be configured to hold swap space and some, or all, of the vendor’s Unix imple- mentation. This scheme is easy to administer, because the local disk should only require updating for changes to the system configuration or new releases of the operating system. However, much of the traffic on a typical network involves access to home directories and locally-maintained software. These data change more frequently and are much larger and hence more difficult to manage when they are distributed across local disks. Many installations attempt to store some local software and user files on workstation disks, but the choice of which data to hold is often arbitrary and measurement of the actual effectiveness is difficult.

By monitoring the NFS traffic from a group of workstations, it is possible to categorize the filesys- tems being accessed and predict the traffic savings that could be obtained by storing different filesys- tems on a local disk. This allows the effectiveness

1992 LISA VI - October 19-23, 1992 - Long Beach, CA

of simple disk allocation schemes to be compared. For certain types of data (read-only program and data files), a more detailed analysis, at the individual file level, shows which files are being accessed most frequently and the savings that could be obtained by holding specific subsets of these files locally. By examining the access times of such files, it is possi- ble for a program to regularly re-configure a local filesystem so that it always holds an optimal subset.

NFS Traffic Analysis

The Department of Computer Science network at the University of Edinburgh includes several hun- dred heterogeneous workstations spanning several subnets. Each subnet is self-contained in terms of workstation services, such as booting, base operating system and local software, but the automounter AMD [3] provides a global namespace that presents a uniform filesystem across all the workstations, using NFS as the remote access protocol[4, 5].

A typical selection of workstations on a single subnet was chosen as a representative sample for the purposes of traffic analysis. These consisted of 26 assorted Sun workstations running SunOS 4.1, classified as follows/:

personal 23 personal workstations, 4 with their own swap disks. public 1 public multi-user machine with a local

disk holding the base operating system and local swap space.

compute 2 compute servers with their own disks, used for swapping only. These workstations serve a computer science

research laboratory, running applications such as text processing, mail, and program development. The

1The fileservers on the subnet are not included, since the

traffic to the servers will be exactly that traffic generated by the workstations.

Effective Use of Local Workstation Disks in an NFS Network

environment is similar to many academic and research installations, and, although there may be significant differences in usage patterns for teaching or commercial networks, many of the following results should still be appropriate.

The programs rpcspy and nfstrace [6] were used to collect data on all NFS transactions from the above workstations over a period of one month. These data were analyzed by several perl [7] scripts that divided the traffic into the following categories by examining the NFS filehandles present in the tran- sactions:

home Access to user’s home directories. local Access to locally maintained software.

root Access to the workstation’s root parti- tion.

usr Access to the workstation’s usr partition. swap Access to the swap file.

misc Miscellaneous access to remote filesys- tems. This includes print spool direc- tories, network-wide temporary direc- tories and remote traffic to other domains.

As would be expected, the amount and distribu- tion of traffic between the categories varies consider- ably between individual workstations, and from day to day. This is owing to the changing nature of the individual’s work; a particular user might even be absent for a week, leaving a workstation idle. How- ever, it is the average figures that reflect the total impact of all the workstations on the network and the servers.

misc

home

root local

Figure 1

On the diskless workstations, swapping traffic accounted for an average 30% of the total NFS traffic during the month. For individual workstations, this varied between 10% and 40%, obviously depending on the type of work being performed by the user, and on the real memory available”. It was notice- able that users with heavy compute requirements are not necessarily those that generate most swap traffic,

Mostly 16Mb in this sample.

Anderson

since they often tend to use remote compute servers for large jobs.

The public machine and the personal worksta- tions showed a roughly similar distribution of the remaining traffic (Figure 1), although the balance between local software and home directories varied considerably; in the extreme case of the two com- pute servers, these two values were actually reversed. This appears to be due to compute- intensive programs accessing large data files from user’s home directories, and paging off users own program binaries.

Standard Configurations for a Local Disk

A conventional allocation of space on a local workstation disk would probably include swap, root and usr filesystems.

In general, local swap space is almost always effective, although the actual amount of disk space required, and the amount of traffic saved on any individual workstation, will vary greatly. The above situation is probably typical, representing an average traffic saving of around 30%, with a swap space of 20-30Mb.

Local storage of the root filesystem is also efficient, occupying about S5Mb of disk and saving 8% of the traffic, but the total saving is small and the use of local root partitions is more difficult to manage.

In those situations, where users rely largely on the software supplied with the vendor’s operating system, local storage of the entire usr filesystem could produce a substantial saving in network traffic; this may also provide some degree of self-sufficiency for the workstation in the event of a server failure. In many large installations, however, users require access to a much wider range of software; in the above example, most users rely on local software even for their shells and window systems, so a local copy of the usr would only generate a saving of about 10% for 130Mb of disk space, and would not provide any useful degree of resilience.

Public Files and Software

Network wide public programs and read-only data files (such as fonts) account for a large propor- tion of the network traffic. For the above sample, most of this traffic is concentrated on the separate filesystem holding locally maintained software (local), although for many installations, a larger pro- portion might be generated from the standard usr filesystem. However, both these filesystems have exactly the same characteristics and it would seem reasonable to hold local copies of frequently used public programs and data files, whatever their origin. Such files are not normally written to by the works- tation and it is usually sufficient to update the local copies from some master copy at relatively

1992 LISA VI - October 19-23, 1992 Long Beach, CA

Anderson

infrequent intervals - for example nightly, or simply on demand. This seems to be common practice and several schemes have been documented, using widely available software[8, 9, 10, 11]. In general, however, there will be far too much software avail- able on the network for copies of everything to be held locally (our network has over 700Mb of local software and 130Mb in usr), so some subset of the available software needs to be selected. A number of schemes have been proposed[10, 12, 13, 14] for organising master copies of the software so that sen- sible subsets (usually corresponding to individual applications) can be extracted and selectively copied onto a local disk. Some of these, such as Depot [12] and /fu[10] include software which can replace cer- tain subsets of files with symbolic links. This would allow a local filesystem to store actual replicas of some files, and links to remote copies of other files, thus permitting files to be transparently migrated on and off the local disk by changing links into file copies and visa-versa.

100 g 80 . 60 r fo ___personal g 40 public S ' compute < 20 Prcmmaecee od , | 0 5 10 15 20

Cache size (days)

Figure 2

This type of scheme has the potential to be very effective; for our network sample, Figure 2 shows the percentage of local traffic? which could be saved by holding copies of just those files accessed in the last N days. These results are surprisingly consistent across the different types of workstation and show that nearly 90% of network traffic from the local software occurs to files that have been accessed in the last two weeks; it seems that only a small fraction of the total available software is actu- ally in use at any one time.

Figure 3 shows the local disk space required to hold all files accessed in the last N days, implying that 90% of the Jocal traffic on a personal worksta- tion can be saved for only 40Mb of disk space, pro- viding that the appropriate files can be chosen.

3Local software has been emphasised in this discussion because it forms a much larger proportion of the traffic in our network sample, but there is no reason to expect that the results from the usr filesystem would be significantly different.

1992 LISA VI - October 19-23, 1992 - Long Beach, CA

Effective Use of Local Workstation Disks in an NFS Network

Unfortunately, the optimal fileset for local storage can be very different on different workstations and obviously varies with time; this is reflected in the larger quantities of disk space required by the public machine and the compute servers, which are used by many different people. Making a reasonable choice manually is very difficult; for example, out of the hundreds of fonts for the window system, each user will have a personal preference and will use only a small, but different, fraction of those available.

Cache Size (Mb)

15 20

5 10 Cache size (days)

Figure 3

File Caching

The above results suggest that a file caching scheme which stored only the recently used files on a local "cache" disk, could be very effective. Other work on general-purpose file caching schemes [15] has confirmed the effectiveness of this technique, but it is unlikely that any general-purpose cache could be effectively implemented without extensive modifications to the kernel filesystem. However, this particular category of files has a number of pro- perties which simplify the problem considerably, and many of the necessary mechanisms are already in place:

@ Immediate reflection of master changes into the cache is not normally critical.

@ The local copies are normally read-only as far as the workstation is concerned.

@ A file can be moved out of the local cache, simply by replacing it with a symbolic link to a network-mounted copy of the file. Con- versely, the a file can be encached by replac- ing the link with an actual copy of the file.

@ If one of the available distribution schemes is already being used, then cache re-arrangement can conveniently occur at the same time as the local files are updated from the master.

The only remaining problem is to select the optimal set of files to be held in the cache, and for the software distribution program to re-arrange the local disk so that these files are held locally, and other files are replaced with links back to the net- work copies.

Effective Use of Local Workstation Disks in an NFS Network

It should be quite possible for the selection of the optimal fileset to be performed independently of the cache update, and this is probably the cleanest solution. However, if the update program is already traversing the filesystem to identify out-of-date files, it is very easy for the same program to collect infor- mation about the previous usage of these files, and this is the approach adopted below.

A Caching Version of the /fu Program

The Computer Science department network uses nightly runs of the /fu[10] program to update copies of the local software onto multiple servers and onto the larger workstations. This program sim- ply performs a selective copy of files from the mas- ter directories (mounted over NFS) onto the local disk. Those files which have been updated on the master are re-copied, and manually selected subsets of files can be blocked, or replaced with symbolic links to copies elsewhere on the network.

To investigate the possibilities of an automatic caching scheme, /fu was modified to:

® Collect information on the previous usage of the files and links on the local disk.

© Apply an algorithm to each file, computing a cache score intended to reflect the desirability of holding a local copy of the file.

@ Re-arrange the local disk to hold as many files as possible with the highest cache score (replacing other files with symbolic links).

100 _ 80 & 8 60 Cc : Ze @ 40 [personal —~ é public -——-

20 | ..-.comput

1 i; | 0 50 100 150 200 250 300 350 400 450 500 Cache overflow (%)

Figure 4

As each file is examined, the "last access time" (atime) of the file is used to record whether or not the file has been used since the last update. This allows a running history of the file usage over the last two weeks to be constructed and maintained on the disk. For an infinite-sized cache, this would be sufficient and the 90% saving indicated by the simu- lations could be expected simply by holding local copies of all files used in the last two weeks. In practice, the cache disk has a fixed size, and there needs to be some way of deciding which files to remove from the cache if it becomes full. Two perl scripts were used to perform full simulations of the cache behaviour, on the NFS trace data, to

Anderson

investigate the performance of different fixed cache sizes, with different algorithms; least-frequently-used and least-recently-used. The daily inspection of the file atime provides only a very coarse measure of the file usage, but the following results show that this is more than adequate in practice; Figure 4 gives the simulated performance degradation, for different lev- els of cache overflow, using the least-frequently-used algorithm (100% represents the performance of an infinite cache).

The trace-driven simulations show a very simi- lar performance for the two algorithms, indicating that the most frequently used files are also the most recently used files, and that the choice of algorithm is not critical.

The least-frequently-used algorithm was imple- mented in /fu, with the additional option of locking certain files into the cache, or varying the priority applied to the cache score. A number of personal workstations and a Sun 690/MP compute/multi-user machine were configured to run this caching scheme. The workstations were allocated 7O0Mb and the 690/MP, 120Mb. This includes the 40Mb (90Mb) indicated by Figure 3, and 30Mb of overhead for the directories and symbolic links needed to reference the 43,000 files on the master. Figure 5 shows the real performance of the cache on the 690/MP.

100 80

60

Hit rate (% files)

Figure 5

In this case, the data were gathered from the live /fu program and shows the percentage of dif- ferent files, on each day, which were accessed from the cache. Although this is not directly comparable with the simulation data (which shows the percen- tage of blocks transferred) it provides an indication that the cache is performing as expected. Examina- tion of the actual files being decached, shows that the sizing of the cache is also approximately correct; most files being removed from the cache having been used on no more than one day in the last two weeks.

Some Practical Issues

A nightly update of the cache generates some network traffic that is not included in the above

1992 LISA VI - October 19-23, 1992 Long Beach, CA

Anderson

figures; this could be a problem for sites running jobs that require significant amounts of network bandwidth during the night. However, the bulk of this traffic would already be generated by any con- ventional distribution scheme and is concerned with identifying files that need to be updated. Examina- tion of typical /fu updates (which are not particularly efficient), using etherfind, [16] shows that about 1KB of network traffic is generated for each file scanned, yielding a total of about 40Mb for the 40,000 files on the sample system? and the extra traffic necessary to adjust the cache, amounts to only an additional 5-10%. Careful scheduling of the nightly updates (which normally take about half an hour each) can usually avoid network congestion and interference with other background jobs, such as backup.

Holding only the most recently used files, means that the local disk provides no real resilience against server failure; certain very important files (such as files required only at boot time) may not be used regularly enough to appear in the cache. The graph in Figure 2 does indicate that a reasonable degree of resilience may not be achievable without holding copies of all possible files, since there appears to be a constant 10% of "new" files used, on average, each day. A more successful solution to this problem is to provide several alternative servers carrying complete sets of all the files (This redun- dancy is comparatively easy to arrange, using AMD). It is possible however, to "wire down" the contents of the cache at any point (for example, after a reboot), or to manually specify files which should be locked in the cache for resilience purposes.

The coarse-grain measure of file usage can sometimes cause the cache to be "flooded" by a sin- gle unexpected access to a large number of infre- quently used files. For example, a user running the command "grep FOO *"in /usr/include will cause the atime of all files in the directory to be updated, possibly raising their cache score above more regularly used files®. The use of a Jeast- frequently-used algorithm (rather than Jeast- recently-used) and a threshold of two days usage for initial entry into the cache, prevent nearly all of these problems in practice.

Several technical problems prevent the atime of files from always giving a reliable indication of the file usage. In particular, the atime of continuously running programs is not continuously updated, and the atime of files which are exported over NFS is not always correct. Symbolic link atimes are also

Clearly, if a lot of files require updating because the masters have changed, then this figure will increase.

This can be observed at several points in Figure 5, which displays the cache hit rate as the percentage of different files, rather than network traffic. The low hit-rate around day 71 was generated by a user browsing a large number of rarely used fonts.

1992 LISA VI - October 19-23, 1992 Long Beach, CA

Effective Use of Local Workstation Disks in an NFS Network

updated rather too easily; for example, by the com- mand "ls -1". /fu uses a number of heuristics to deal with the worst of these cases and they do not appear to be a problem in practice.

Home Directories

Home directory access is responsible for a large percentage of the traffic from most workstations. If these directories are held on a central fileserver, con- siderable savings could be expected by moving them onto a local disk. For the above sample, Figure 6 shows the percentage of this traffic that could be saved by transferring user’s home directories onto their own workstations, and the amount of disk space that would be required in each case®.

* (b) personal public +

Average saving (%)

0 50 200 250 300

100 150 Disk space (Mb) Figure 6

In most cases, virtually all of the home directory traffic can be eliminated by allocating about 50Mb of disk space to the user’s home directory”. How- ever, this has a number of drawbacks:

@ The amount of space required is difficult to predict and may vary greatly between users. In the above example, one user (b) requires much more than 50Mb and many users require much less. Reallocation and efficient use of this space is difficult.

@ The directory must still be available from elsewhere on the network which means that the workstation must export the filesystem over NFS. Integrating filesystems from large numbers of different workstations and arrang- ing backups is more difficult to manage.

@ The savings are greatly reduced in any situa- tion where a user regularly works on more than one machine; for example, in a pool of ten compute servers, or public machines, which are used at random, the saving will be less than 10%.

SUsers without their own workstation are transferred

onto the public machine.

The one workstation showing a particularly low saving in the above example (a) represents a machine that was being used by several different users for a particular package with node-locked licensing.

Effective Use of Local Workstation Disks in an NFS Network

When a caching scheme is being used for pub- lic software, then it may be beneficial to encourage users to export copies of their own software along with the other public software. This is especially useful if the software is heavily used and/or fre- quently used by other people, when it provides the performance benefits of caching, together with resili- ence for other users of the program.

Some Possible Developments

Most existing distribution schemes work well when they are updating small numbers of files on a large number of machines, or a large number of files on a small number of machines. The above caching scheme is appropriate for most workstations with their own disk, however small the disk, so it makes unusual demands on the distribution system by requiring very large numbers of master files to be scanned by every workstation, even though the number of files actually transferred is usually very small. This scanning process is by far the most costly factor in the nightly update operation and some type of update program which could simultane- ously broadcast the information to several clients would considerably reduce server and network load.

The current implementation links only files, and always copies directories. This means that there may be very large directory hierarchies, that are never accessed by a particular machine, but still occupy significant amounts of space on the local disk for the directories themselves and the symbolic links that they contain. In theory, such a hierarchy could be replaced by a single symbolic link to the remote root of the hierarchy, but in practice, it is difficult (perhaps impossible) to identify individual workstation access to files in such a hierarchy and to arrange for its expansion when a decision is made to encache one of the files that it contains.

Conclusions

Clearly there will be some variation in usage patterns between sites, that will affect the optimal configuration for local workstation disks. There are also a number of trade-offs to be made; for example, the trade-off between efficiency and ease of manage- ment involved in the placement of home directories. The degree to which traffic can be saved® by caching a small number of well-chosen files on a personal workstation is, however, surprising and a number of conclusions would seem to be generally applicable:

@ Holding swap and root partitions on a local disk is generally effective.

@ Holding full copies of the usr filesystem may not be worthwhile in many cases.

8Note that this does not necessarily imply an improved performance for every workstation; network access to fast disks on a lightly loaded server can be faster than slow local disks.

Anderson

@ Caching of public files is very efficient in terms of reducing the network and server load. This does involve additional administration and nightly network bandwidth, but not much more than existing schemes which update fixed sets of files onto a local disk.

@ Holding local copies of home directories is normally effective only for personal worksta- tions, or machines that are used significantly more often by an individual user. It is likely to be difficult to match space requirements to available disk space though, and some compromise such as providing the user with both a remote, and a local directory may be necessary.

For a typical personal workstations from the survey, the following configuration would be an effective way of organising a local 15O0Mb disk:

@ 25Mb swap, saving 30%

@ SMb root, saving 5%

@ 70Mb local cache, saving 90% of 41% = 36% @ 50Mb home, saving 15%

Thus providing a total saving of over 80% of the network and server load generated by the works- tation.

Optimization of local disks on public machines and compute servers is more difficult; caching of public files is still effective, but the home directory traffic becomes a more significant proportion of the total traffic and this is more difficult to place and less effective than on a personal machine.

Acknowledgments

Thanks are due to Matt Blaze for the use of the rpcspy and nfstrace programs, and to the system staff in the Computer Science Department for their suggestions and assistance during development and testing of the /fu program.

Availability

The caching version of /fu is available for anonymous ftp from ftp.dcs.ed.ac.uk in the directory pub/1lfu. The documentation file in this directory also includes the references[10, 4, 5].

References

. Edward R Zayas, ‘‘AFS-3 Architectural Over- view,’ in AFS-3 Programmer’s Reference Manual, Transarc Corporation, September 1991.

2. R Sandberg, D Goldberg, S Kleiman, D Walsh, and B Lyon, ‘‘Design and implementation of the Sun Network File System,’’ Proceedings of Usenix Conference, Summer 1985.

3. Jan-Simon Pendry, AMD An automounter, Department of Computing, Imperial College, London, May 1990.

4, Paul Anderson and Alastair Scobie, ‘‘The Local

UNIX directory _hierarchy,’’ ©CS-TN-21,

e

1992 LISA VI - October 19-23, 1992 - Long Beach, CA

Anderson

Department of Computer Science, University of

Edinburgh, Edinburgh, August 1991.

. Paul Anderson, ‘‘Installing software on the Computer Science Department network,’’ CS- TN-24, Department of Computer Science, University of Edinburgh, Edinburgh, August 1991.

6. Matt Blaze, NFS Tracing by passive network monitoring, Department of Computer Science, Princeton University.

7. Larry Wall, Perl -— Practical Extraction and Report Language.

8. Sun Microsystems, ‘‘rdist (1),’’ in SunOS refer- ence manual, Sun Microsystems, 1990.

9. Steven Shafter and Mary Thompson, The SUP software upgrade protocol, Carnegie Mellon University, September 1989.

10. Paul Anderson, ‘‘Managing program binaries in a heterogeneous UNIX network,’’ Proceedings of LISA V Conference, pp. 1-9, Usenix, 1991.

. Bjorn Satdeva and Paul M Moriarty, ‘‘Fdist: A domain based file distribution system for a heterogeneous environment,’’ Proceedings of LISA V Conference, pp. 109-126, Usenix, 1991.

12. Wallace Colyer and Walter Young, Depot: A tool for managing software environments, Car- negie Mellon University, April 1992.

13. John Sellens, ‘‘Software maintenance in a campus environment: the Xhier approach,”’ Proceedings of LISA V Conference, pp. 21-28, Usenix,