Collection evaluation and the conspectus

Abstract

Although collection measurement has been much practised by librarians, it is seldom linked to precise, qualitative and client-centred objectives. The RLG conspectus offers a methodology whose validity for Australia needs to be examined carefully. Program performance evaluation has already proved illuminating at the State Library of New South Wales. Collection evaluation is useful for cooperation if properly documented.

The first step is to measure whatever can be easily measured. This is okay as far as it goes. The second step is to disregard that which can’t be measured or give it an arbitrary quantitative value. This is artificial and misleading. The third step is to presume that what can’t be measured easily isn’t very important. This is blind-ness. The fourth step is to say what can’t be easily measured really doesn’t exist. This is suicide. ¹

Why evaluate?

The body of literature on collection evaluation is consider-able. In 1977, Lancaster, in his landmark work The measurement and evalu-ation of library services, stated that more work had been carried out on the evaluation of collections than on any other aspect. ² The American Library Association’s Guidelines for collection development included a section on “Guidelines for the Evaluation of the Effectiveness of Library Collections”, and the methods and procedures outlined became entrenched as a part of library life. The various reasons for conducting evaluations of collections have been well documented. The ALA Guidelines stated:

Evaluations should be made to determine whether the collection is meeting its objec-tives, how well it is serving its users, in which ways or areas it is deficient, and what remains to be done to develop the collection. ³

Mosher’s seminal work outlined the benefits of collection evaluation rang-ing from use in policy development to justification for book budget increases. ⁴ Gorman and Howes, in their recent excellent treatise on collection develop-ment, draw attention to the use of evaluation as a guide for collection plan-ning, budgeting and decision making, determining whether a library is achieving its objectives and satisfying its users. They summarize the various reasons for collection evaluation in presenting eleven aims, nine of which are those cited by Mosher in 1979:

1.	To search for more accurate understanding of the scope, depth and utility of collections.
2.	To prepare a guide and a basis for collection development.
3.	To aid in the preparation of a collection development policy.
4.	To measure the effectiveness of a collection development policy.
5.	To determine collection adequacy or quality.
6.	To help rectify inadequacies in library holdings and to improve them.
7.	To focus human and financial resources on areas most needing attention.
8.	To aid justification for book budget increases.
9.	To demonstrate to administrators that something is being done about the demands for “more money”.
10.	To establish the existence of special strengths, as well as weaknesses, in the collection.
11.	To check the need for weeding and collection control, and to establish areas of priority of need. 5

Other writers have detected a shift in the thrust of collection evaluation. McLachlan and Trahn, while addressing the areas already identified by previ-ous writers, indicated that in their evaluation of the University of New South Wales collection in 1982, their task was begun largely as a response to finan-cial stringencies and a consequent need for more effective cooperation, a need to determine effective utilization of available funds, and a need to document areas of regional or national significance in the collections. ⁶ The impetus for Swinburne Library’s collection evaluation was provided by its shrinking storage space. ⁷

The reasons for conducting collection evaluations are wide-ranging and appear to be situational, related to the specific needs of particular institu-tions. In addition, the focus has primarily been on internal use of data col-lected, particularly in relation to the preparation of collection development policies, and collection evaluation as a basis for cooperation has been a recent phenomenon. Other questions on collection evaluation need to be put to add to an understanding of the reasons.

How effective are current approaches to collection evaluation?

The term “evaluate” is frequently used quite loosely and indiscriminately. Many of the activities included within collection evaluation constitute collection descrip-tion and measurement, rather than evaluation. The results are frequently bib-liographical and statistical in nature. Without entering the debate on the use of quantitative or qualitative measures, many of the methods for collection evaluation involve the checking of lists or the statistical analysis of circula-tion statistics. Wainwright noted that evaluation involves ascertaining the value of a collection; a subjective concept with ideas of quality, rather than just measurement. ⁸ Other writers on collection evaluation have referred to notions of quality, but few attempts have been made to identify or define quality. The tendency has been to do what was referred to in the opening quotation - measure whatever can be easily measured. Measurement is useful for identifying collection gaps but does not address rectifying the situation unless targets are also specified. This approach to collection evaluation focuses on the past, rather than looking to the future.

The various approaches to collection evaluation have been described as collection-centred or client-centred. One example of a collection-centred approach is the work done on random samples of items listed in bibliogra-phies to determine holdings of a specific library. Another example is the com-parison of the titles listed in an indexing service with the holdings of a specific library. Without a benchmark or target percentage, such data alone reveal little about the quality of the collection. For example, in an examination of our serials holdings at the State Library of New South Wales, we have examined the percentage of titles held of various Wilson indexes and other services. The percentage of titles indexed in Social sciences index and held by the Library is 350Jo. Should this figure be 400Jo, 600Jo or 800Jo? A number of evaluation programs have used interlibrary comparisons to gain some idea of quality. The University of New South Wales Library in its assessment holds 500Jo of titles indexed in Social sciences index. While one could say that the State Library should at least try to attain the same percentage, I am still not certain that either of us, by this measure, can assume anything about the quality of our collections.

The approaches to evaluation known as collection-centred might perhaps be more strictly defined as librarian-centred. While some evaluations, par-ticularly in academic settings, have sought opinions from users about the extent of coverage of a specific subject area, most evaluations have been con-ducted by librarians. A library collection exists only for the use of its present and future clients, and approaches based on client use appear to be more useful in making quality judgements than those focused solely on the collec-tion itself. At the State Library of New South Wales, we have used several client-centred approaches. An analysis of our interlibrary loans from March-June 1988 showed that 350Jo of the items requested were not held. Again a benchmark is required-should this figure be lOOJo? In this instance, com-parisons with other libraries would not be useful, given that, since the introduction of charges for loans, NSW academic libraries have agreed not to charge public libraries if their requests are first channelled to us. Collec-tion use has been monitored at the State Library of New South Wales in the last twelve months through an analysis of our stack request slips, using a personal computer and a variety of database management, graphics and statistical software packages primarily from the public domain. ⁹ Several ana-lyses have been carried out, in particular a categorization of use by subject. The following table illustrates this use.

Again, what does this tell us about the quality of the collection? If the social sciences are receiving heavy use, does it mean the collection is satisfy-ing demands? We did a further analysis of the breakdown of our collection holdings by subject. Should there be a match between the percentage use of the collection and the percentage holdings in a specific area? I have already mentioned the possible use of interlibrary comparisons. Coincidentally, the National Library did a similar study at the same time. The similarities between the two are inescapable. Can the differences, however, be ascribed to qual-ity differences in the collection? A closer examination of the data leads to the possibility that the differences in the 300s might relate to collection strengths at the National Library in education and in the 600s to collection strengths in medicine.

These examples illustrate the difficulties in current approaches to collec-tion evaluation which focus on measurement and do not adequately address the objectives of specific collections, particularly in relation to client serv-ice, and do not develop target performances in relation to quality. Before leaving this topic of the effectiveness of current approaches, I would like to make some comments on the effectiveness of the RLG conspectus as a means of collection evaluation.

How effective is the RLG conspectus as a collection evaluation tool in the Australian environment?

There appears to have been some confusion in the Australian discussions to date on the use of the RLG conspectus. The conspectus consists of a very detailed list of subject areas coupled with col-lecting levels which are then allocated to those subject areas. Some people using the term conspectus have been referring to the collecting levels only.

The collecting levels used in the conspectus and the subject areas of the con-spectus are not necessarily interdependent. The levels can be applied to other types of subject description for collection analysis and policy development. Likewise other types of levels could be allocated to the subject areas identi-fied in the conspectus.

In the event of the RLG conspectus being used in Australia, the possible means of recording the data collected has not been clarified. The data could be recorded by some libraries in-house as part of their own collection develop-ment processes. The data could also be stored centrally. The use of the RLG conspectus has developed in the US into the North American Collections Inventory Project. The real value of central storage of information and its possible uses in the Australian context would need careful identification and agreement to justify the considerable costs involved.

Eric Wainwright’s paper alludes to the collecting levels we have developed at the State Library of New South Wales. When we began policy formula-tion in 1986, the RLG conspectus and levels were examined. Agreement had not then been reached by the Committee of Australian University Librar-ians on the use of the RLG levels. The reasons for the choice of levels at the State Library of New South Wales are outlined in the Draft collection development policy. ¹⁰ To summarize, four levels have been chosen. These are defined in relation to the nature of enquiries rather than educational criteria, largely because of our client base. In addition, the term comprehen-sive has not been used because it is considered that with the range of resources we collect e.g. manuscripts, pictures, computer software, comprehensiveness is neither desirable nor attainable.

The subject areas in the RLG conspectus were also examined in our policy formulation. It was decided that a less detailed subject breakdown based on Dewey would best suit the libraries with whom we wished initially to cooper-ate, i.e. public and government department libraries. The RLG conspectus is based on the Library of Congress classification which is not used by most of our clients. A number of approaches were examined, including those in various states of the US, e.g. Illinois. A data collection sheet for use at the State Library of New South Wales was devised, based on one used in the state of New York.

Most of the criticisms of the use of the RLG conspectus have been out-lined in the literature, although songs of praise outweigh the dirges of carp-ing critics. The critics argue that the conspectus is too imprecise to be instructive, too untestable to invoke belief, and too laborious ever to repay the effort. ¹¹ There are several features of the Australian environment which must be carefully considered, and which I believe detract from the suitabil-ity of the conspectus for this continent, or at the very least, change its application.

One factor in the Australian environment I would like to consider is the existence of the Australian Bibliographic Network (ABN). The use of the conspectus focuses on collection evaluation at the macro level. Collection evaluation at the macro level is of use in collection development, but its benefits to the client are more difficult to identify, particularly if one defines the quality of a collection in relation to its ability to satisfy client needs. A researcher in Australian history of the nineteenth century would no doubt benefit from knowing that the State Library’s Mitchell Library collection is of great excellence in this area and a visit to its premises would facilitate access to the contents of the collection. However, many researchers and library clients operate at a micro level - for them the quality of a collection is evalu-ated on the basis of its ability to supply a particular item. This kind of access is provided by ABN. I would prefer to see Australia’s scant resources put into improving the data recorded in ABN and the inclusion of retrospective holdings, rather than in further description at a macro level of library holdings.

Other factors in the Australian context inhibiting the use of the conspectus relate to regionalized use of resources, the small number of research libraries, and the multiplicity of other types of libraries.

A new perspective?

I have called into question the current approaches to collection evaluation. Is there indeed anything new to add? Wainwright in his 1984 paper on collection adequacy registered surprise that few new ideas on collection evaluation had been brought forward since the 1970s.¹²

For many of us, the concepts of strategic planning, performance, effec-tiveness, efficiency and appropriateness have become synonymous with public sector management in the 1980s. The emphasis on accountability of public sector organizations has been implemented within finance and annual report legislation and most government jurisdictions within Australia stipulate man-datory monitoring of programs, although this monitoring may take differ-ent forms throughout the country. These new approaches may shed some light on the collection evaluation process.

At the risk of being parochial, I would like to outline some of the develop-ments in NSW. These developments do mirror activities in other states. The introduction of program performance evaluation review by the NSW govern-ment has changed our approaches to evaluation considerably. Program per-formance evaluation is the process of making judgements about the appropriateness, efficiency and effectiveness of a program- its objectives, client needs, procedures, activities, resources, management strategies and out-comes. A program can be defined as any set of procedures, activities, resources and/ or management strategies designed to achieve some common goals or objectives.

In answering the question “Why evaluate?” in this context, apart from the mandatory reasons, the purpose of the evaluation is to inform managers of the relevance or otherwise of a particular program, to highlight inefficiency and waste, to modify or discontinue a program, to assist in planning a pro-gram, to further its implementation, and to assess the impact of any changes. The evaluation process involves the identification of outcomes. As far as the collection is concerned these outcomes must relate to use.

Once these have been determined, activities related to each outcome are identified, as are factors affecting their successful achievement. Performance indicators are developed in relation to the objectives. Collection evaluation in this scenario then emphasizes the purpose of the evaluation, the establish-ment of outcomes and targets, an examination of the activities and use of performance indicators.¹³

We have recently completed an evaluation of our serials program. Effec-tive use of our serials collection is the desirable outcome. To achieve this, people must be aware of serials as a source of information. Our studies have shown that our activities relate more to the collection and maintenance of serials, rather than to raising awareness of their information content. A number of performance indicators can be used, e.g. percentage use of serials in relation to monographs and cost comparisons.

We have found the approach to the performance evaluation review process most illuminating. Its emphasis on a program which cuts across all depart-ments has been beneficial. Because collections and means of access, either intellectual or physical, are inextricably interwoven, I believe that evalua-tion of a library program, rather than evaluation of the collection per se is more useful. For example, the State Library is currently evaluating its Com-munity Language Lending Service. It is impossible to examine the content of that collection without examining also the means of provision and the type of listing, i.e. cataloguing, which makes its contents available.

This approach to performance evaluation makes the purpose clear. Its end is improvement, and changes which are required occur during the process of the evaluation. Its focus on objectives and outcomes provides a clarity of vision. I believe that these approaches can provide the new perspective on collection evaluation which is required.

Is collection evaluation a prerequisite for cooperation?

The final aspect of collection evaluation I would like to touch on is its relationship to coopera-tion. I am presenting a state library perspective. The need for state libraries to cooperate with public and government department libraries has already been mentioned. Each state’s legislation, structure and objectives in relation to these libraries varies, but all state libraries act to back-up local resource provision, through loans or cooperative purchasing agreements. Our draft collection development policy is being used as a basis for discussion with our various stakeholders, and in particular, to determine whether the col-lecting levels allocated to specific areas are appropriate. State libraries cooper-ate with each other and with the National Library of Australia in the collection of “stateiana” and “Australiana”. At the State Library of New South Wales, we focus on our own state, and leave the rest of Australia to the other state libraries, with the National Library taking an overall perspective and a preser-vation role. The legislative deposit requirements vary from state to state but generally speaking a copy of items published in each state is deposited with the state and National libraries. Even here cooperation is neeeded and we have held discussions with the National Library and informal agreements exist with regard to ephemeral materials and serials considered to be of lesser significance. There is also cooperation in the enforcement of the deposit legis-lation. Commitment to cooperation rather than collection evaluation will lead to success in these areas.

State libraries and the National Library have traditionally been the libraries in this country which have collected manuscripts, pictorial materials and ephemera. Much of this cooperation is based on known strengths e.g. material on Flinders at the State Library of New South Wales. Informal agreements exist in this area, and the record on cooperation has been reasonably good. The entry of other players to this arena, namely academic libraries, has proven that cooperation is much more difficult to achieve where objectives vary con-siderably. Many libraries pay lip service only to cooperation. All libraries must focus initially on their own client base, and its needs, and all are look-ing for a place in the sun.

Many discussions have been held on “core” collections. Most large libraries will contain the same base material which generates most use-the 80/20 rule. In looking at areas for cooperative collection development, it is the areas outside this core, the periphery, where agreement on collecting needs to be obtained. Any approach to cooperative collection development should be based on decisions about this material. Libraries then need to describe their collecting intentions according to level in specific subject areas. Most will collect to basic in all subject areas, and choose those areas for advanced treat-ment according to local client need and library objectives. Cooperation on these areas would then achieve generalized in-depth coverage of subject areas throughout Australia and prevent the situation where, as happens at the moment, several libraries cancel simultaneously the same serial title which is peripheral to their interests. There are many success stories in coopera-tion. One of these is the Sydney Subject Specialization Scheme where public libraries take responsibility for joint coverage of all subject areas. Agree-ment by academic libraries on cooperative collecting is difficult to achieve because of changing research and teaching interests. State libraries cooper-ate with other research libraries. For example, the State Library of New South Wales is a member of the Office of Library Cooperation. Cooperation in collection development on a small scale has been achieved, but until collec-tion development policies are in place for each institution, indicating where a collection is going, rather than where it has been, then it is difficult to see what can be achieved.

Conclusion

In summary, I have argued that collection evaluation as it has been practised can more accurately be described as collection measurement. It is nonetheless useful as a basis for cooperation provided that there is a commitment to future directions in collecting by individual libraries and that negotiation on responsibility for collecting in peripheral areas is undertaken. What is essential to effective cooperative collection development is the exis-tence of written collection development policies in major libraries. I have questioned the validity of the RLG conspectus in the Australian context, although elements of its approach are useful in arriving at appropriate col-lection descriptions in individual libraries, particularly as a basis for collec-tion development policies. I have suggested that new approaches to collection evaluation are necessary so that individual libraries can determine the over-all effectiveness, efficiency and appropriateness of their activities. These approaches include focusing on program objectives and outcomes, the deter-mination of performance measures and standards or targets to be achieved, and an emphasis on program improvement as the prime purpose of evaluation.

I have also indicated that evaluation reviews which include aspects of col-lection organization and use, as well as content, will facilitate program improvement. Successful cooperation depends on commitment-where there’s a will, there’s a way. What we really need to achieve cooperative collection development in this country is a greater sense of commitment, a willingness to negotiate and to hold to agreements, and a vision of what can be achieved. To offer my personal paraphrase on the quotation which opened this paper,

The first step is to measure whatever can be easily measured. This is okay as far as it goes. The second step is to determine the direction in which we wish to go. This is positive and forward thinking. The third step is to cooperate with others and negotiate. This is wise and farsighted. The fourth step is to implement what has been decided and evaluate the outcomes and correct deficiencies. This is life-renewing and will secure the future.