| $3861 | Data analysis by ISSR | |
| $840 | Room rental | |
| $335 | Photocopies | |
| $4530 | Incentives | |
 | ||
| $9566 | TOTAL | |
Spring 2001
UCLA Library Instructional Services Advisory Committee
Patti S. Caravello, Chair
Eloisa Gomez Borah
Judith Herschman
Eleanor Mitchell
Librarians have long had anecdotal evidence that undergraduates do not possess adequate information skills for some of the coursework they are required to complete. To obtain an objective measure of their information competence, the UCLA Library’s Instructional Services Advisory Committee (ISAC) conducted an assessment project. The committee created a list of competencies and a survey instrument, which was administered to a sample of 453 undergraduates in Spring 1999. This report explains the research problem and methodology, explores the findings and conclusions of the research project, and makes recommendations based on the data.
The main goal of the project was to identify ways to make library instruction more effective at UCLA. A practical objective was to obtain data to use in discussions with faculty about students’ information and research skills, the impact of those abilities on students’ coursework, and the potential of library instruction to improve them.
The instrument ISAC created sought to measure how skillful or knowledgeable students were in general with library resources, online searching, and information-seeking concepts, rather than to assess the efficacy of existing library instructional programs. The instrument was vetted in several ways over the course of the project and was administered in a non-course-related classroom setting to a broad sample of students. A data analyst from the Institute for Social Science Research (ISSR) oversaw the coding of data and performed several types of analysis on it to test hypotheses and verify significant findings.
Results indicate many gaps in students’ understanding of resources and methods, which are discussed in detail in the report. The general level of information literacy as assessed by the instrument was low. Statistically significant findings based on an analysis of average scores and student demographics are:
The results did not allow ISAC to identify causes for these findings, although a number of hypotheses are possible. It is not clear whether or in what way the statistically significant results are substantively significant.
The mean scores of students who reported having had a high quantity of library instruction or tours were also analyzed. Although these students did not score significantly higher on the test, two thirds of them had their library classes or tours in high school; the number who had the sessions in college was so small that the result for this variable is not particularly meaningful.
Based on the results of this project and the collective experience of committee members, ISAC has made several recommendations aimed at the goal of increasing information competence. Library staff should share the key findings with faculty and create a dialogue about the information competence of their students. This might include discovering how faculty view students’ information skills and exploring the effect of increased library use on information competence. Library staff should work with faculty and academic departments to define, adopt, and promote sets of basic and discipline- or major-oriented competencies; these should be used to develop library instruction that is part of a curriculum-integrated information literacy program. The Library should take a more systematic approach to instructional initiatives – which may include Web-based instruction, course-integrated instruction, courses, and remote learning mechanisms. Areas for further research and recommendations about the use of the instrument are also included in the report.
The Instructional Services Advisory Committee appreciates the support and patience it received from the UCLA Library administration, particularly Janice Koyama and Rita Scherrei, who generously funded the project. Thanks are also owed to Sonia Luna for her help in distributing the financial incentives. ISAC acknowledges the UCLA Registrar’s Office for distributing the email to a random sample of undergraduates. The committee is grateful to Chris Corey of the Institute of Social Science Research for the data analysis and assistance he provided and to the 453 UCLA students who took the time to complete the questionnaire. Thanks are owed to Dawn Setzer for her editorial assistance. Former committee members who contributed to the project are Susan Allen, Kathy Dabbour, Marie Waters, and Amy Atchison.
Based on information from instructors and from their own experiences, librarians have long had anecdotal evidence that undergraduates do not possess adequate information skills for some of the coursework they are required to complete, let alone for lifelong learning. UCLA’s approach to teaching undergraduates these basic skills through library instruction has been vigorous but decentralized, and basic information literacy goals for undergraduates do not currently exist. Therefore, to obtain hard data on the information competencies of UCLA undergraduates that will further a campus-wide discussion regarding establishing and implementing basic information literacy goals for all undergraduates, the UCLA Library’s Instructional Services Advisory Committee (ISAC) embarked on an assessment project supported by the Library administration.
There are many articles on information literacy, lists of competencies, and descriptions of information literacy programs and courses. However, there is a paucity of actual assessment tools that measure student competencies rather than evaluate library instruction. For example, pre-test/post-test measures assess the efficacy of particular library instruction sessions but do not test general information competence, and user surveys rating satisfaction and describing library use do not provide an objective skills assessment. Thus, one of the problems to be addressed in this study was the creation of an assessment tool.
To create a tool, the committee first had to decide on a list of competencies. Although this study was completed before the publication of the ACRL Information Literacy Competency Standards for Higher Education,[ 1 ] the list of competencies it measures was derived from a review of articulations by colleagues in the profession as well as from discussions among ISAC members, who represented libraries across campus. The competence lists in print and on academic library Web sites are too numerous to cite; however, several Web sites from the California State University libraries are worth noting because of the enormous amount of information literacy work they have done here in California.[ 2 ] Lorie Roth describes information competence assessment initiatives at several universities in California, including a study at California State University, San Marcos, to measure student attainment in information literacy over a four-year period.[ 3 ] Susan Carol Curzon describes the California State University Information Competence Skills Assessment, a telephone survey in which 3,000 students responded to hypothetical research and information-seeking scenarios.[ 4 ]; the results of this survey have not yet been published.
Other academic libraries have undertaken related assessments and have developed their own instruments, which were of interest. An effort to compare student self-assessment of information skills with objective evaluation was described by Greer, Weston, and Alm.[ 5 ] Two surveys at Johns Hopkins University compared freshmen and upperclassmen at that institution, focusing first on evaluating basic library skills and then adding some advanced skills.[ 6 ] The Indiana University Bloomington Libraries Assessment Plan for Information Literacy, while not offering a single assessment tool, describes a plan for assessing information literacy and articulates specific objectives and possible measurement techniques for each information literacy goal.[ 7 ] Kent State librarians report on a pretest used to assess basic library skills primarily of freshmen enrolled in an English II course.[ 8 ] The Teaching Library at University of California, Berkeley, first surveyed graduating seniors in the political science and sociology departments in March-May 1994 and has repeated the survey with students in these and other disciplines[ 9 ], Pat Maughan is completing an article summarizing the results.
The committee hoped that the results would provide an impetus for campus-wide discussions on information literacy needs and goals. After realizing that the information literacy literature contained so little on assessment of competence, ISAC also determined that this study would contribute to the understanding of information literacy assessment.
The overall goal was to assess whether exposure to the library – over the course of an under-graduate’s years at UCLA, through the requirements of a major, through library instruction, or simply due to a higher rate of library use – has any impact at all on student information competence. Several indicators of exposure were explored, and hypotheses were that:
The committee hypothesized in the broader sense but did not venture hypotheses on particular competencies (e.g., whether students were competent at online searching with Boolean operators, whether they know when it is appropriate to make a footnote, etc.). However, the project did produce information on particular competencies, which are reported in section IV.F.
The committee defined the following information competencies that all UCLA graduating seniors should possess. The numbered items on the list are the competencies; the lettered items are the specific behaviors and skills a student needs to employ effectively in order to achieve that competence.
Information competence of undergraduate students at UCLA is defined by the ability to:
The goal was to administer the questionnaire to 500 undergraduate students; the final sample size was 453. The Registrar’s sample was randomized by systematic sampling so every undergraduate had an equal chance of being picked. The response rate (calculated as the number of students who took the test divided by the number who received the email) was 14.3%. It was assumed that the number of students who actually opened the email or opened it in time for one of the test dates would be lower than the number who received the email. A lower number in the denominator would have yielded a higher response rate, but there is no way to know how many opened the email. How to increase the response rate remains elusive.
The data about the sample in Table 1 are derived from the demographic portion of the instrument. The data about UCLA undergraduates as a whole are derived from the UCLA Campus Profile for 2000,[ 12 ] which gives data for 1999. With the low response rate, it is not possible to say this sample was representative of the entire UCLA undergraduate population, even though the sampling technique was randomized. Still, the percentages for sex and major are fairly similar.
Table 1: Characteristics of the Sample and of UCLA Undergraduates as a Whole
| Sample | UCLA Undergraduates | |
| Sex | ||
| Female | 61% |
55% |
| Male | 39% |
45% |
| GPA[ 13 ] | 3.1 |
N/A |
| Class | ||
| Freshman | 28% |
15% |
| Sophomore | 24% |
20% |
| Junior | 18% |
29% |
| Senior | 30% |
37% |
| Major | ||
| Social Sciences | 36% |
28% |
| Sciences | 34% |
41% |
| Humanities | 8% |
10% |
| Arts | 4% |
3% |
| Unknown or Undeclared | 18% |
16% |
Before administering the test, the committee filed for an "Exemption from Human Subject Protection Committee Review" through the UCLA Office for Protection of Research Subjects. The exemption was granted on April 23, 1999 (UCLA Exempt #99-091). This process is required of all campus research projects using human subjects.
Participation was solicited and the test was administered as follows:
ISAC spent a good deal of time reviewing advice from various experts and reading articles in order to come to a decision on how best to administer the instrument. The committee weighed the advantages, disadvantages, and expense of multiple mailings of the instrument with and without telephone follow-up and considered privacy issues related to posting it on the Web or sending it via email. In the end, certain facts were evident. It is not an interesting or enjoyable test to take, it lacks humor, and it solicits no personal opinions on topics of concern to students. Regardless of how it was administered, without an incentive (and even with one) there was concern about obtaining a sample of sufficient size. Some committee members also felt that having the students take the test in a classroom setting, where there was no possibility of consulting others or the computer to answer the questions, would be a better test of what they knew at that moment in time.
This study produced "information on groups and phenomena that already exist" and created no control or comparison groups. It therefore has a "descriptive," or observational, design. Specifically, it is a "cross section" study, providing descriptive data at one fixed point in time.[ 14 ] Participants will not be retested at a later date as no coding for tracking them was built in, as would happen in a cohort study.
The Library administration supported the project; specifically, the associate university librarians for public services and for human resources allocated the funds to complete it. The amount of money spent was $9,566, broken down below. This represents cash outlays and in no way accounts for the amount of person-hours spent over the three-year period.
| $3861 | Data analysis by ISSR | |
| $840 | Room rental | |
| $335 | Photocopies | |
| $4530 | Incentives | |
| ||
| $9566 | TOTAL | |
The 25-item, self-administered questionnaire (in Appendix A) consists of:
The instrument does not test the effectiveness of particular library instruction sessions or whole programs, nor is it a survey of opinion. It is a test to discover how skillful or knowledgeable students are generally with library resources, online searching, and information-seeking concepts. For this study, the test was administered on paper in a non-library classroom setting. Students took about 10-20 minutes to complete it.
Focusing on the list of competencies and members’ knowledge of undergraduate information needs and behaviors, the committee created, discussed, and revised the test questions. Each question was keyed to one or more of the competencies on the list. A draft of the instrument and the competencies list was then shared with other librarians in various library units, who were asked to key the questions to the competencies, if they could; the idea was to see if their matches and those of committee members would be the same. As a result of this valuable exercise and the suggestions it produced, some questions were revised and changes were made in some of the question-competence matches. The exercise also gave the committee a better sense of both the content validity and face validity of the instrument. This process of front-line librarians creating an instrument and additional librarians with fresh eyes providing feedback on how well the questions tested the competencies seemed to be the best method at the committee’s disposal. The only published survey the committee referred to in any significant way during this process was from Johns Hopkins (1993).[ 15 ]
When a solid draft of the instrument was completed in February 1998, it was tested on a group of nine students who work in the Research Library’s Access Services Department. Based on information from the test results and verbal feedback solicited through the process, several revisions were made to the instrument.
In October 1998 a more ambitious pilot test of the revised instrument was conducted on a convenience sample of three large lecture classes.[ 16 ] Blank tests were distributed to 683 students, which they were told to complete on their own time and return at the next class session; the instructor did not require the students to take the test, and 127 completed samples were returned. Although these samples were not scored, a question-by-question analysis was conducted to determine how many and what percentage of students got each question right, wrong, or partially correct. This enabled the committee to see which questions might be problematic. Based on the results and further refinement of the questions, some wording was changed and reformatting done to improve the test’s look, content, and the location of instructions.
Finally, in April/May 2000 (well after the May 1999 study was completed), the instrument was sent by email to students in the library school, asking only second-year students to complete and return it either by email or to the YRL reference desk. No incentive was offered. From a pool of approximately 70 second-year students, 16 completed samples were returned (a response rate of 23%). The goal was not to improve the instrument, which had already been used to test undergraduates, but rather to compare the average scores of the undergraduates with those of a group of "expert" subjects, i.e., students expected to do well. The mean score of the library students (86.8%) was significantly higher than the mean score of the undergraduates tested (61.7%). This increased the committee’s confidence in the study and provided a basis for comparison with which to characterize the undergraduate scores and answers.
Through analysis of variance it was possible to compare the mean score of students who had a high rate of library use[ 17 ] (N=188) with the mean score of students who had a low rate of library use (N=226). The average score of the high library use students (64.2%) was higher than the average score of the low library use students (59.7%), and the difference is statistically significant, that is, it was not due to chance, nor was it due to other factors (class or major). Since these scores appear fairly close, however, it is not certain that the difference indicates something meaningful.
It is interesting to look at some of the specific findings within the library use data. A slightly greater difference in scores than the overall one was seen between the high and low users of ORION, Melvyl, and journals and books. This is shown in Table 2.
Table 2: Mean Scores by Level of Library Use [ 17 ]
N |
Mean Test Score | |
| Library Use (all variables) | ||
| 188 | 64.2% | |
| 226 | 59.7% | |
| ORION Use | ||
| 199 | 64.7% | |
| 254 | 59.2% | |
| Melvyl Use | ||
| 141 | 65.9% | |
| 307 | 59.7% | |
| Use of journals, books | ||
| 148 (jour) 167 (books) |
65% | |
| 303 (jour) 284 (books) |
60% |
Analysis of variance and t-tests indicated that the average score of seniors (66%) was statistically significantly higher than the mean scores of any of the other classes. This finding was not due to chance or other factors (major or library use). There was no significant difference between the mean scores of freshmen, sophomores, or juniors. Comparisons were made class by class as well as between each class and everyone else. Although seniors scored higher, for the group tested there was no trend indicating the higher the class level the higher the score. From this data it is not possible to tell whether the statistically significant six-percentage-point-difference between seniors and the others is substantive.
Class |
N | % of Sample | Mean Test Score |
Freshman |
124 | 28% | 59.9% |
Sophomore |
109 | 24% | 59.5% |
Junior |
82 | 18% | 59.9% |
Senior |
137 | 30% | 66% |
Total |
452 | 100% | 61.7% |
There were 33 majors represented by the 373 students who reported majors (80 reported no major or "undeclared"). The 33 majors were recoded into four main groups because there were too few students in each major to make comparisons meaningful. Table 1 shows that the students in the sample were not evenly distributed between the four groups, but they roughly corresponded to undergraduates in general at UCLA. Students whose major fell in the area of the humanities scored statistically significantly higher on the test (with a mean of 66.3%) than students in the social sciences (62%), arts (63.9%), and sciences (60.2%). This finding was not due to chance and was independent of other factors (class, library use).
However, again the differences are not great enough to conclude that there is a substantive significance to the finding. From the data it is not evident whether the hypothesis that "students in disciplines that require independent library research will score higher than students in disciplines which do not" is valid. While humanities students may well do more independent library research than science students, it is not possible to tell if the differences here are due to this, nor are the differences great.
Table 4: Mean Scores by Major Area
Major Area |
N | % of Sample | Mean Test Score |
Humanities |
37 | 8% | 66.3% |
Arts |
16 | 4% | 63.9% |
Social Sciences |
164 | 36% | 62% |
Sciences |
156 | 34% | 60.2% |
Unknown/Undeclared |
80 | 18% | N/A |
The students who reported having had a high quantity[ 18 ] of library instruction or tours anytime from high school through college did not score significantly higher on the test. However, almost two thirds of the 52 students in the "high" group had their library instruction or tours in high school, where most of the skills and concepts tested by this instrument would probably not have been covered. The number of students in the "high" group who had their library instruction in college was so small (N=19,[ 19 ] or 4.2% of the sample) that it is impossible to say that this result indicates anything meaningful in the university setting. Table 5 shows how many and what percentage of students fell into the "high" library instruction group.
The very small number of students who had had more than five instruction sessions or tours at UCLA (N=3) is a reflection of the fact that library instruction at UCLA is not systematically infused into the curriculum. If library instruction were broadly integrated into courses and objectives for majors, it is likely more students would have been in the "high" library instruction group, and the survey results might have been informative regarding the relationship between library instruction and information competence; as it is, the small numbers do not allow any conclusions to be drawn. Section V.C. below on further research includes a suggestion on this.
Table 5: Students in the "High" Library Instruction Group
Variable |
N | % of Sample |
Whole course (v11) |
14 | 3.1% |
Over 5 at UCLA (v12) |
3 | 0.7% |
Over 5 at another college (v13) |
2 | 0.4% |
Over 5 in High School (v14) |
33 | 7.3% |
Total |
52 | 11.5% |
The test questions were grouped into five clusters corresponding to the five main information competencies. The data was analyzed to see how many students got all the questions correct in a single cluster in order to find out what percentage of students were knowledgeable enough to answer correctly all questions about a set of skills as opposed to individual skills. Hardly any students succeeded in doing this. The fifth competence ("Organize and synthesize") was covered with only one test question, so the cluster approach was not meaningful for that competence. The highest number of students getting all correct in a given cluster (not counting competence five) was the eight students who got all three questions (eight variables) correct in the "Develop a search strategy" cluster. The negligible number of students getting all correct in any of the first four clusters indicates sporadic knowledge rather than competence. This conclusion is supported by the generally low scores of the undergraduates. Table 6 contains the data on the question clusters.
| Competence | Test questions in cluster | Variable numbers | Percent of students answering all questions in cluster correctly |
| 1. Define the topic | 1, 5, 8, 12 | v31, v49, v56, v66 | 1.3% |
| 2. Develop a search strategy | 2, 3, 10 | v32, v33-38, v58 | 1.8% |
| 3. Locate & retrieve information | 4, 6, 9, 11 | v39-47, v50, v57, v59-65 | 0.4% |
| 4. Evaluate information & strategy | 13, 14 | v67-75, v77 | 1.5% |
| 5. Organize & synthesize | 7 | v51-54 | 35.8% |
Only 18% of the sample did not know how to proceed if their online search result was too big; 82% knew to add search terms and try again. Yet 45.5% did not know that in a Boolean statement, OR retrieves more records than AND or NOT. While 68% knew to change their search terms if they retrieved zero results on "French revolution," one-third of the sample did not know what to do in that case. The mixed results here (reflecting competencies 2c and 3a) perhaps indicate sporadic knowledge as opposed to solid competence.
About half (52%) the respondents said they would check a statistic from a newspaper in a government source before using it in a term paper, meaning the other half would simply use the unverified data or check it in the prior year’s newspaper. A more intriguing result is that although 84% of the students thought the date and author of the Web site would help them evaluate the authority and accuracy of the information provided on the site and about half thought the Web address (including "arco.com") would help them evaluate it, two thirds (66%) did not include the link to "What ARCO does" as an element that would help evaluate the site’s authority and accuracy. Most of the students correctly did not check off the parts of the Web site which would not help in this regard (variables 71, 72, 74, and 75).[ 20 ] Yet 67% said that the Web site created by ARCO would be valuable as an objective source on which to base a paper on air pollution in Los Angeles. If the student was unaware of what ARCO is, the "don’t know" option would make sense, but only 5% chose this option. Only 28% correctly said it would not be a valuable resource for this purpose. (Among the library school students, 75% said it would not be a valuable resource.) Oddly enough, although 15% said they did not know which elements of the Web site would help them evaluate its authority and accuracy, only 5% said they did not know if the site would be valuable.
About 62% of the sample could not identify a correct and complete journal article citation for a bibliography. Although the "wrong" choices given were not extremely wrong, they did lack the entire article’s page numbers and they included extraneous information; moreover, the correct answer was not dependent on knowing a particular style of citation. A similar number, about 60%, did not identify as a journal article one of the article citations in the question 11 reference grid, and 40% could not identify the other one. Between 55-58% thought the three book citations were journal articles or other forms or checked "don’t know." This stands in contrast with the 97% who correctly identified the Web site.
In addition to the ability to decipher bibliographic references (competence 3b) is the intellectual process of citation (competence 5b). In question 7 three instances are given, all of which would require a footnote in a research paper. Ten percent of the students either checked "don’t know" or said none of these instances required footnotes. Slightly over one-third of the students would not include a footnote if they copied a whole paragraph in their paper. And 56% would not include a footnote in a research paper if they read an article and wrote it over in their own words. But 76% knew that if they quoted a sentence from the article they should include a footnote. These results again indicate sporadic knowledge rather than solid competence.
Although only 11% of the sample checked "don’t know" on what information can be discerned from a call number in question 4, it is clear many more did not know. About one quarter of the students said the publication date could not be determined (although the example included one), and about the same percentage said the number of pages can be determined. About a quarter said the owning library can be discerned from the call number, and a slightly smaller percentage said the location of the item cannot be determined from the call number. For each of the variables just mentioned (publication date, number of pages, owning library, location), about three-quarters of the respondents knew what was discernible from the call number. Still, 60% did not think they could tell which cataloging system the library uses by looking at the call number. And fully 72% did not indicate that the subject can be discerned from the call number; this stands in comparison with the library school students, all but one of whom knew that the subject can be discerned. The results here (competence 3d) speak to familiarity with and understanding of a very basic aspect of libraries which most librarians take for granted that all undergraduates already know.
Fully 78% of the sample said that searching the Web would be the best way to identify current and authoritative information for a research paper on the Y2K problem; only 15.5% would use a periodical index. Two-thirds (67%) of the students could not identify the primary source on the list in question 8. To find the borders of the former Yugoslavia, 38.6% would correctly use the Encyclopedia Americana, but 35% would use the Atlas of American History or the Encyclopedia of Associations. The average percent correct for all the variables in question 3, in which students had to choose the best source for each topic, was 61.4%. (The library school students’ average for these variables was 90.7% correct.) Once again, the undergraduates were aware of some good strategies and resources, but many of the results are disappointing.
As with much research, answers lead to more questions. The randomly selected group of under-graduate students tested did not display a high level of information competence as measured by the instrument. The average score of 61.7% is quite low relative to the average score (86.8%) of the group of library school students. The statistically significant differences noted for seniors, humanities majors, and students who use library resources frequently are intriguing. If there is no gradual rise in scores for the four classes, why did the seniors score highest? What is it about majoring in the humanities that led to higher scores? The data do not answer these questions. Speculation as to why the high library-use group did better includes: more experience with the online systems, more trial and error, and more exposure to information-seeking problems and solutions may lead to somewhat more knowledge. "Statistical significance, however, need not have anything to do with substantive interpretations of the factors, since a statistically significant factor may not always be identified correctly in terms of empirical phenomena."[ 21 ]
It has to be stated again that the differences between the groups, although not due to chance alone (by definition of "statistically significant"), were not large enough on the face of it to be truly impressive. The difference between 66% and 60% is small when one considers that the higher score means an average of 31 correct out of 47 variables, while the lower score means 28 correct out of 47 variables. The committee cannot necessarily say that the study’s statistically significant findings "reveal something meaningful about the object of study,"[ 22 ] i.e., are substantively significant. Yet they do bear out to varying extents three of the four hypotheses, which were in turn based on librarians’ observations and experiences.
The statistically significant findings are based on average scores of groups defined by class, library use, major, and past library instruction in order to test the hypotheses. The 453 students’ individual scores ranged much more widely than the average scores of the groups studied (from 27%-89%), but the data analyses do not explain the differences between the highest and lowest scores. Might there have been other ways to define groups? Perhaps by grade point average, ethnic or language factors, gender, even personality characteristics related to information-seeking or education? As these are factors librarians cannot affect, they were not examined. They nevertheless might be worth investigating in order to understand information competence better; thus, these conclusions lead directly into recommendations for further study.
A close look at the findings from the clusters, the individual questions, and the overall scores leads to the conclusion that while students are not at a total loss when it comes to the concepts and skills tested, at best they possess sporadic knowledge. The librarians on the committee had considered the test an easy one from the outset; had more difficult questions been included and a larger number of questions testing each competence in more depth, the scores would probably have been even lower.
With respect to the finding related to library instruction, it is significant that library instruction at UCLA is decentralized and so not, in an overall way, geared toward an established set of information literacy objectives nor characterized by systematic curricular tie-in. Individual sessions incorporate objectives and curricular relevance, but there is no progressive, formal instruction sustained through the four years or as part of the major; library instruction is not formally included by acacias departments or university administration as a required part of a UCLA education. Given that, and despite the hypothesis that the "high" library instruction group would do better, it is not wholly surprising that they did not. If this group had been larger, would the result have been the same? It is not possible to tell. Nonetheless, a new hypothesis arises from this finding: that undergraduates whose colleges require or incorporate systematic, curriculum-integrated information literacy education, with all the goals, ways, and means in place, will be more information competent than students in schools which lack such a focus. It is possible that what makes the difference is not just the number of library instruction sessions a student has had, but whether they are part of an integrative program.
The instrument was necessarily neutral in terms of academic discipline; yet assessment, like information literacy education, might best be accomplished in the context of particular disciplines, "as information literacy manifests itself in the specific understanding of the knowledge creation, scholarly activity, and publication processes found in those disciplines."[ 23 ] It is impossible to know if the students would have done better if the questions had related solely to their major subjects; the committee’s assumption is that the basic or general competence the instrument sought to assess would still be important in order to do well on a more subject-oriented test.
Based on a careful review of question responses, the scores and other data from the questionnaires, and members’ combined experience, the committee recommends:
In addition to these recommendations, the committee has ideas for further research. Some of these ideas arose from members’ experience of conducting the research and using the instrument, others from analyzing the results.
Download questionnaire in Adobe PDF (157KB)
Download questionnaire in MS Word for Windows (157KB)
As a sidelight of the study, one of the committee’s hopes was to learn about information literacy assessment and to contribute something to the literature on it. It is therefore appropriate to record some of the strengths and limitations of the instrument developed for this project.
Although it would have been preferable to include more questions on the test part of the instrument in order more thoroughly to cover particular competencies, if it had been any longer or more tedious to take, there would certainly have been fewer volunteers. It is a difficult balance to achieve. As it is, some competencies were covered with two or three questions, others with one, and some with none ( 2d, 3c, 5a). This was sometimes because a testing instrument does not lend itself to assessing certain competencies.
For example, number 5a, "critically use and integrate information from a variety of sources appro-priate to the research question" was, the committee felt, impossible to test with this sort of instru-ment. The best test of that competence might be a review (by the professor) of a term paper to see whether there was a critical and analytical approach to information from appropriate and varied sources. A test item for this might have had the student review a piece of writing and answer questions about its use of information and resources (not exactly the same thing as his writing the piece himself), but this would have made the test far too time consuming to take.
For the 453 completed questionnaires, scores ranged from 27% to 89%, which appear as a bell-shaped (or "normal") curve on a graph (see Appendix C). When an analysis of "predictive association" was run, a few items (4 out of 44 variables) were found not to be predictive. That is, getting the right answer on the question was not predictive of information competence (as defined by a high score on the test). When the four variables that fell into this category were examined carefully, three of them did not pose a concern; in other words the committee did not believe that these were somehow bad questions, only hard ones. The fourth non-predictive item (question number 1 on the test, which was variable 31) gave the committee much more concern because it could be seen as too subjective a question from a certain viewpoint. When every student in the "expert" group from the library school got this question wrong, committee members were finally convinced it was a bad question. If the test were to be given again, this question would have to be rewritten.
In the demographic portion of the instrument, a question should have been included asking how many research papers the student had done. It would perhaps have been a more meaningful measure of "library use" than some of the other items in question 11 (v22-30).
On the whole, the short multiple choice test has its limits for assessing a large and complex set of knowledge and skill. The suggestions for further study (section V.C) offer ideas for completely different types of testing, where the student would actively have to demonstrate competence with research strategy concepts and tools rather than passively pick from given choices on a test.
------------- TO: (student name) FROM: Patti S. Caravello, Librarian, UCLA Young Research Library DATE: May 10, 1999 RE: $10 for your thoughts How would you like to receive $10.00 for 15-20 minutes of your time and help improve the UCLA Library’s services at the same time? You are among a small group of UCLA undergraduates I am asking to fill out a short questionnaire on information resources and research skills. You will be paid $10.00 cash when you hand in the completed questionnaire. Here’s all you need to do:
1 Association of College and Research Libraries, Information Literacy Competency Standards for Higher Education (Chicago: ALA, 2000).
2 California State University Information Competence Work Group, Information Competence in the CSU: A Report, 1995 http://www.calstate.edu/LS/Archive/info_comp_report.shtml (September 14, 2000); California State University Information Competence Work Group, Information Competence, 2000 http://library.csun.edu/susan.curzon/infocmp.html (September 14, 2000); California State University Information Competence Assessment Project, 2000 http://www.csupomona.edu/~kkdunn/Icassess/ictaskforce.html (September 14, 2000); California State University Fullerton Pollak Library, Assessing Information Competence Across Disciplines, 2000 http://users.library.fullerton.edu/infocomp/index.htm (September 14, 2000).
3 Lorie Roth, "Educating the Cut and Paste Generation," Library Journal 124 (November 1, 1999): 42-44.
4 Susan Carol Curzon, "Developing a Program of Information Literacy," College & Research Libraries News 61 (June 2000): 483+.
5 Arlene Greer, Lee Weston, and Mary Alm, "Assessment of Learning Outcomes: A Measure of Progress in Library Literacy," College and Research Libraries 52 (November 1991): 549-57.
6 Jill Coupe, "Undergraduate Library Skills: Two Surveys at Johns Hopkins University," Research Strategies 11 (Fall 1993): 188-201.
7 Indiana University Bloomington Libraries Assessment Planning Committee, "An Assessment Plan for Information Literacy," May 1, 1996, http://www.indiana.edu/~libinstr/Information_Literacy/assessment.html (September 19, 2000).
8 Lilith R. Kunkel, Susan M. Weaver, and Kim N. Cook, "What Do They Know?: An Assessment of Undergraduate Library Skills," The Journal of Academic Librarianship 22 (November 1996): 430-34.
9 Pat Davitt Maughan,, "Information Literacy Survey, " University of California, Berkeley Library, 1995, http://www.lib.berkeley. edu/TeachingLib/Survey.html (September 19, 2000).
10 An earlier version of the first goal reflected the committee's original desire to compare the competence of freshmen with that of seniors. The survey design was later changed for logistical reasons, and the first goal was altered to reflect that change by referring simply to "undergraduates."
11 The second goal reflects the committee's hope at the project's outset that the study would give library staff information that could be used to help improve library instruction. In no way did the committee view the study or the instrument as capable of measuring the effectiveness of library instruction at UCLA.
12 UCLA Office of Academic Planning and Budget, "Student Data" in UCLA Campus Profile for 2000, 2000. Tables used: "Fall 1999 Actual Enrollment by Sex and Level," "Enrollment by School and College," and "1998-99 3-Quarter Average Enrollment By Department, By Level of Student." http://www.apb.ucla.edu/prof00.htm (July 25, 2000)
13 The average GPA given for the sample is the average for the 439 students who supplied this information. Average GPA for all UCLA undergraduates was not available.
14 Arlene Fink, The Survey Handbook, The Survey Kit, v. 1 (Thousand Oaks, CA: Sage Publications, 1995), 25.
15 Jill Coupe, "Undergraduate Library Skills: Two Surveys at Johns Hopkins University," Research Strategies 11 (Fall 1993): 188-201.
16 The classes were GE 20A, Political Science 10, and Sociology 1.
17 "High library use" was defined as using specific library resources such as online catalogs ORION and Melvyl, reference books, journals, books, reserve lists, the Web, etc., four or more times since fall quarter and visiting a library to use its resources at least once a week. The data came from variables 20-30 on the instrument.
18 A "high" quantity of library instruction was defined as having taken a whole course in library research methods and resources or having had a library class or tour more than five times in high school or college.
19 Fourteen of the 19 reported having taken a whole course in library research methods. Although responses do not indicate whether it was at UCLA, another college, or in high school, these 14 are grouped with the five known to have had their library instruction in college.
20 A fair percentage of the students (27% for v71 and v74, 14% for v72, and 42% v75) thought that the links to the information offered by the site would help them in the task of evaluation. While it is understandable to want to see the information to be able to judge it, a more critical approach would lead to the conclusion that it hardly matters what the information is when the author cannot be seen as an objective or reliable source of information on the topic for a research paper.
21 Peter H. Rossi, James D. Wright, and Andy B. Anderson, eds., Handbook of Survey Research (San Diego: Academic Press, 1983), 274.
22 W. Paul Vogt, Dictionary of Statistics and Methodology: A Nontechnical Guide for the Social Sciences (Newbury Park, CA: Sage Publications, 1993), 226.
23 Association of College and Research Libraries, Information Literacy Competency Standards for Higher Education (Chicago: American Library Association, 2000), 6.
This is the first publication of the UCLA Library’s Information Literacy Initiative, which has been created to enhance the Library’s efforts to instruct students in information literacy skills and to encourage and inform a campus-wide dialogue on this subject. To be placed on the mailing list for future publications, please send your name, mail and email addresses, and telephone number to: Dawn Setzer, Director of Library Communications, UCLA Library, 53442 Young Research Library, Los Angeles, CA 90095-1575 or email <dsetzer@library.ucla.edu>.
Copyright © Regents of the University of California
Persons wishing to reprint any portion of this report are required to write in advance for permission.