DOCUMENT RESUME
ED 478 598
HE 036 021
AUTHOR
Beile, Penny M.; Boote, David N.; Killingsworth, Elizabeth K.
TITLE
Characteristics of Education Doctoral Dissertation
References: An Inter-Institutional Analysis of Review of
Literature Citations.
PUB DATE
2003-04-00
NOTE
25p.; Paper presented at the Annual Meeting of the American
Educational Research Association (Chicago, IL, April 21-25,
2003).
PUB TYPE
Reports
Research (143) -- Speeches/Meeting Papers (150)
EDRS PRICE
EDRS Price MF01/PCO2 Plus Postage.
DESCRIPTORS
Citation Analysis; *Citations (References); *Doctoral
Dissertations; *Graduate Students; Graduate Study; *Scholarly
Journals
ABSTRACT
This study had two purposes: to examine the expertise of
doctoral students in their use of the scholarly literature and to investigate
the use of citation analysis as a tool for collection development. Analysis
of 1,842 coded citations gleaned from 30 education dissertations awarded in
2000 from 3 institutions in the United States revealed that journal articles,
at 45%, were cited most frequently, followed by monographs (33.9%) and
"other" (18.3%), with magazines and Web sites contributing less than 2% each
of the total material types cited. The study examined 858 journal and
magazine citations, which were found in 239 unique titles. A relatively small
number of journals contained a high percentage of the references found in the
dissertations analyzed. Based on a design by D. Kohl and L. Wilson (1986),
dissertation citations were also scored for scholarliness, currency, and
appropriateness of format, and scores on the three criteria were averaged to
arrive at a quality rating. Results of interinstitutional comparisons
revealed a significant amount of variation and were considered in conjunction
with institutional characteristics and published criteria for quality
bibliographies. The data suggest that the assumption of doctoral student
expertise in their use of the scholarly literature may be overstated and
should be examined in relation to their preparation for professional status.
For purposes of developing a library's research collection, a core list of
titles, generated on the basis of multiple, rather than single, institutional
analysis is indicated. (Contains 8 figures, 11 tables, and 28 references.)
(Author/SLD)
Reproductions supplied by EDRS are the best that can be made
from the original document.
Characteristics of Education Doctoral
Dissertation References: An Inter-
Institutional Analysis of Review of
Literature Citations
Penny M. Bei le, David N. Boote, and Elizabeth K. Killingsworth
U.S. DEPARTMENT OF EDUCATION
Office of Educational Research and Improvement
EDUCATIONAL RESOURCES INFORMATION
CENTER (ERIC)
13
This document has been reproduced as
received from the person or organization
originating it.
O
Minor changes have been made to improve
reproduction quality
a Points of view or opinions stated in this
document do not necessarily represent
official OERI position or policy.
2
1
PERMISSION TO REPRODUCE AND
DISSEMINATE THIS MATERIAL HAS
BEEN GRANTED BY
P. Bei le
TO THE EDUCATIONAL RESOURCES
INFORMATION CENTER (ERIC)
AST copy Mk
Characteristics of Education Doctoral Dissertation References, p.
1
Paper summary for AERA Annual Conference
2003
Teaching Statistics Roundtable
SIG: Professors of Educational Research
CHARACTERISTICS OF EDUCATION DOCTORAL DISSERTATION
REFERENCES:
AN INTER-INSTITUTIONAL ANALYSIS OF REVIEW OF
LITERATURE CITATIONS
Penny M. Bei le, Associate Librarian
David N. Boote, Assistant Professor
Elizabeth K. Killingsworth, Associate Librarian
University of Central Florida
Abstract
This study had two purposes, to examine expertise of doctoral students in
their
use of the scholarly literature of the field,
and to investigate the use of citation analysis as
a tool for collection development. Analysis
of 1842 coded citations gleaned from 30
education dissertations awarded in 2000 from three institutions in the United States
revealed journal articles, at 45%, were cited most frequently, followed by monographs
(33.9%) and "other" (18.3%), with magazines and Web sites contributing less than 2%
each of the total material types cited. The study examined 858 journal and magazine
citations, which were found in 293 unique titles. A relatively small number of journals
contained a high percentage of the references found in the dissertations analyzed. Based
on a design by Kohl and Wilson (1986)
dissertation citations were also scored for
scholarliness, currency, and appropriateness of format and scores on the three criteria
were averaged to arrive at a quality rating. Results
of inter-institutional comparisons
revealed a significant amount of variation and were considered in conjunction with
institutional characteristics and published criteria for quality bibliographies. The data
suggest that the assumption of doctoral student expertise in their use
of the scholarly
literature may be overstated, and should be examined in relation to their preparation for
professional status. For purposes of developing a library's research collection a core list
of titles generated on the basis of multiple, rather than single, institutional analysis is
indicated.
Characteristics of Education Doctoral Dissertation References, p.
2
Christine Barry (1997) writes that successful doctoral students
tend to be
"comprehensive and up to date in reviewing the literature," and
accordingly, their
dissertations offer an abundance of bibliographic information.
This articulates a
fundamental assumption that as the doctoral dissertation is
the capstone to the formal
academic training process, associated bibliographies are
high quality, comprehensive in
scope, and reflect emerging
research areas. A limitation of this assumption is the lack
of sound empirical evidence to support it. Very few
studies have been conducted
exploring the quality of dissertation references, and no
studies were identified in the
field of education. Those studies that have investigated the
quality of citations
generally report on such topics as completeness of cited
references (Williams, 1997)
and the increasing use of electronic resources (Davis &
Cohen, 2001; Davis, 2002;
Herring, 2002).
Dissertation citation analysis has frequently been proposed as an
in-house
means to identify journals most important
for the research collection (Buttlar, 1999;
Gooden, 2001; Kriz, 1984; Walcott, 1994; among others). Gooden
(2001) suggests
citation analysis is a useful tool for both eliminating low use journals
and purchasing
needed ones. One potential limitation of relying on dissertation references to create
core journal lists is that the
overwhelming majority of these studies only consider
dissertations awarded by a single institution. Kuyper-Rushing (1999) developed a core
journal title list gleaned from music dissertation bibliographies from across
the United
States and compared them to a single institution's list. She concluded
analysis of a
single institution could result in a skewed list of journals and suggested a
broader
institutional base to arrive at a more objective list of core journals. Without
further
analysis, is it reasonable to conclude, as Gooden (2001) does, that the current
collection
is sufficient for doctoral level research? Or, is it equally plausible to consider
that
students lack the skills necessary to perform an exhaustive review of the literature
and
procure information available external to
the institution?
The role and purpose of the review of literature in the research process can be
found in almost any book on research design and methodology (e.g., Babbie, 1998;
Creswell, 1994; Fraenkel & Wallen, 1996) and journal editors lament
submitted
manuscripts often fail to adequately address the existing body of scholarly
literature
(Grant & Graue, 1999; Hernon, 1994; St. Clair & Hernon, 1996), thus offering that
the
subject is both well defined and of interest to the educational community.
Additionally,
a number of authors have commented upon
the quality of published educational
research (Hall, Ward, & Comer, 1988; Tuckman, 1990; Ward, 1975) and
reported much
of it flawed, of mediocre quality, or otherwise seriously deficient. Although the
review
of literature was only one component of the studies being examined, the review
is
considered an essential part of any reported research.
4
Characteristics of Education Doctoral Dissertation References, p. 3
Only one study was identified that directly addressed the issue of the doctoral
dissertation literature review. Zaporozhetz (1987) reported that not
only did doctoral
students feel their library use skills were inadequate, but their
faculty advisors admitted
they had little knowledge of information retrieval themselves, they
expected their
students to have bibliographic skills at the doctoral level, and they
ranked the review of
literature chapter the lowest when considered in relation to the remaining
standard
dissertation chapters.
The above-mentioned studies focus on the body of published
research in
education, but also have implications for adequacy of professional
preparation. As
such, results of this study will be of interest to professors of doctoral
students,
professors of educational research, and faculty who sit on doctoral
dissertation review
committees. Academic librarians with instruction or collection development
responsibilities will also find this information pertinent.
Similar to earlier studies, this study presumes dissertation citations are
indicative of doctoral students' demonstrated ability to locate and evaluate scholarly
information. However, earlier assumptions of quality of doctoral student review
of the
literature performance are examined by assessing various characteristics of dissertation
citations. Specifically, this study explores the questions:
1)
What are the characteristics of citations in recently awarded doctoral
dissertations in the field of education?
2)
How does a core journal list from a single institution compare to a
list derived from analysis of multiple institutions?
3)
And, what is the relative quality of doctoral dissertation citations?
Method and Data Sources
Defined as a wide-ranging area of bibliometrics, citation analysis studies the
citations to and from documents (Diadoto, 1994), and is one method often used to
generate core lists of journals deemed critical to the research needs of an
institution.
Research studies employing citation analysis methodology are often conducted by
evaluating a sample of citations from student dissertations to develop a core list of
journals, and subsequently, to determine what proportion are locally held and the
estimated strength of the collection (Strohl, 1999).
Thirty education dissertations awarded in the year 2000 from three institutions
in the United States were examined. Each of the institutions offered doctoral degrees in
education, similar acceptance rates to the graduate education program, and a
comparable number of education faculty. Two institutions were purposely chosen for
their similarities in total enrollment (43,000 students in 2000), date of institutional
establishment (mid-1850s), and presence among the top ranked schools of education
("Schools of Education," 2000). The third institution was selected for purposes of
contrast as it was not included in the list of top ranked schools, and enrollment (31,500
in 2000) and date of establishment (mid-1960s) differed.
5
Characteristics of Education Doctoral Dissertation References, p. 4
Dissertation Abstracts and respective institutional library catalogs were
searched to identify all dissertations awarded by the colleges of education
from each
institution in 2000. Results were grouped into the general topic areas
of educational
leadership, educational psychology, instructional or learning theory, and
teacher
education. A purposive sample of ten dissertations across topic areas
and from each
institution was generated and the full dissertation was obtained.
Information extracted from each dissertation included the name of the granting
institution, the total number of citations in the bibliography, the number of
citations
coded, and the number of pages of the dissertation. Citations were
coded by date of
publication, type of material cited, journal or magazine title (if relevant), and
material
format (print or electronic). Types of material consisted of journal,
magazine, Web
site/not electronic journal, monograph, or "other." Examples of items
included in the
category of "other" were ERIC documents, dissertations
and theses, conference
proceedings and presentations, and personal communications.
To address the question of doctoral students' assumed ability to
thoroughly
mine the scholarly information available citations were evaluated on the criteria
of
scholarliness, currency, and appropriateness of the source to the subject being
developed. Based on earlier work by Kohl and Wilson (1986), these criteria were
defined as:
Scholarliness; how good was the fit of the source for the topic? (Did the student
use empirical, peer-reviewed journal articles
rather than accounts in general
magazines? Or, did the student use sources from scholarly presses rather than
popular publishers?)
Currency; was an appropriate decision made regarding retrospective versus
contemporary sources for the topic? (If the student required recent research on
a particular topic were journal articles rather than
books consulted?)
Appropriateness; was the material type appropriate for treatment of the topic?
(If the student needed to develop their rationale for use of a learning theory, was
a book more appropriate than an encyclopedic
entry?)
Dissertations were distributed among three evaluators (one education and two
library faculty), with each evaluator assigned three dissertations from each institution,
plus one additional. The evaluators read the abstract and thesis chapter to familiarize
themselves with the scope and intent of the dissertation and then independently scored
references cited in the literature review chapter. As independent evaluations were
performed inter-rater consistency was tested using a two-way mixed effects model of
the intraclass correlation coefficient in SPSS version 10.0. The average of the scores of
the three evaluators was found to be sufficiently reliable (interval of 0.6766 to 0.9345
with 95% confidence), suggesting that the evaluators were able to successfully and
consistently differentiate among different levels of performance.
Although Kohl and Wilson (1986) scored each of the criteria in their model on a
four-point scale, evaluators in the current study slightly modified their method by using
a four-point scale for scholarliness and a three-point
scale for currency and
6
Characteristics of Education Doctoral Dissertation References, p. 5
appropriateness. The same criteria were applied to both print and electronic
formats.
Data were analyzed at the institutional level and overall.
Descriptive statistics were
generated for dissertation and citation characteristics. Core lists
of journals from each
institution were evaluated for duplicate and unique titles, and then
compared to
institutional holdings to determine the percentage of items locally
available. Both
Kruskal-Wallis and one-way ANOVA tests were conducted examining
differences
among institutions.
Results and Conclusions
Overall, the number of citations coded for this study was 1842. The
total
number of citations per dissertation ranged from a low of 25 to a
high of 159 (M =
87.70, SD = 32.54). As the study was limited to analysis of the
review of literature
only references from this chapter were coded. The number of
citations coded ranged
from18 to 137 (M = 61.40, SD = 32.01). The length of dissertations,
without
appendices, ranged from 76 pages to 329 pages (M = 146.10, SD = 63.06). For
purposes of reporting institutions are
referred to as 1, 2, and 3, with 2 being the
institution of contrast as noted in the previous section. See Table 1 for
dissertation
characteristics by institution.
Analysis of all 30 dissertations revealed journal articles were cited most
frequently, accounting for 45% of citations coded. Journal articles were followed by
monographs (33.9%), and "other" (18.3%), with magazines and Web sites totaling less
than 2% each of the total material types cited. Disciplines vary in their modes of
scholarly communication, and these results suggest that while professional journals
remain the predominant medium for disseminating scholarly information in the field
books and book chapters continue in their importance.
The "other" material type category contained 337 items, or 18.3% of coded
citations. ERIC documents accounted for 35.6% of these materials, followed by
abstracts of dissertations (15.1%), conference papers and presentations (14%), doctoral
dissertations (9.5%), research reports (9%), and law and legislation (6.5%). The
remaining 10.3% were comprised mainly of company reports, email correspondence,
unpublished or submitted manuscripts, policy papers, and master's theses. More than
one in ten of all coded citations were ERIC
documents, doctoral dissertations, or
abstracts of dissertations. The heavy student reliance on and faculty acceptance of
items such as these, that vary immensely in quality, is surprising.
Considerable variation of material type cited was found among institutions.
Notably, dissertations from Institution 1 cited an equal number of journal articles and
monographs (both 43.8%), while the remaining institutions relied more heavily on
journal articles. Also, Institution 2 cited "other" materials much more frequently, at
31.3%, than the other institutions, which were around 10%. See Table 2 for material
type by institution.
7
Characteristics of Education Doctoral Dissertation References, p. 6
In this study, Web sites were differentiated from
electronic journals for
purposes of identifying types of sources,
but not format. Web sites, which accounted
for 1.3% of coded citations, were evaluated on the same
criteria as other citations. In
addition to the 24 Web sites coded another 28 items were
cited as retrieved
electronically for a total of 52 items, or 2.8%, of coded citations. Of
the 52 links,
which were presumably working at the time the dissertations were
written, 28 (54%)
were no longer viable and 24 (46%) were
still extant at the time of this study. Of the
28 electronically retrieved items not coded as Web sites,
nine were journal articles and
the remaining 19 items consisted of email correspondence,
abstracts, law and
legislation, and policy papers and research reports.
Previous research (Davis, 2002; Davis & Cohen, 2001; Herring,
2002; Rusch-
Feja & Siebeky, 1999) suggests that users prefer electronic
information as compared to
print materials. With this in mind, it was unexpected that citations to
electronic
information comprised such a small proportion of the reference list.
Given the access
to full-text electronic articles available at the time the
dissertations were written, it is
plausible candidates retrieved information electronically, yet were not aware
of
conventions for citing electronic information.
Of the 1842 references analyzed 858 were journal and magazine
citations,
which were found in 293 unique titles. Of these, 111 journal citations and 28
magazine
citations (139 total, or 16.2%) were not peer-reviewed. The average date
of publication
for coded journal and magazine citations was 1990 (SD = 7.79). The top 17
journals
accounted for 290, or 33.8%, of the citations coded. The mid-tier, which contained 65
journal titles, returned 309, or 36% of the citations. The remaining 259 citations
(30.2%) were retrieved from 211 titles.
This pattern is consistent with Bradford's Law, which suggests that the
published journal research in a field falls into three zones, each of which includes an
approximately equal number of articles, while the number of journals required to
produce those articles increases substantially from one zone to the next (Wallace,
1989). Essentially, Bradford, and many researchers since, have concluded that a core
number of journals publish an inordinate amount of cited articles (Kriz, 1984; Kuyper-
Rushing, 1999; Radhakrishna, 1994; Summers and Clark, 1986; among others). Table
3 lists the top 17 journals that were cited most frequently overall.
Journal and magazine titles cited were also examined and core lists distinct to
each institution derived. Significant overlap of titles was found among institutions, but
a surprising number of titles unique to individual
institutions were also discovered. Of
the 95 journal and magazine titles cited in Institution 1 dissertations, 56, or 58.9%, were
unique to the institution
Similarly, of the 137 titles cited in Institution 2 dissertations
92, or 67.2%, were cited only by candidates from that institution. Finally, of the 142
titles cited in Institution 3 dissertations 92, or 64.8%, were unique. Tables 4 through 6
list the most frequently cited journal titles by institution.
8
Characteristics of Education Doctoral Dissertation References, p. 7
Similar to Gooden (2001), this study found, across all institutions, that research
collections overwhelmingly contained the sources cited by doctoral students.
Journal
and magazine titles were checked in the online library catalogs of the institutions.
Of
the 196 references cited by Institution 1 candidates, 19, or 9.3%, were not
locally held,
90.7% were owned. Likewise, of the 298 references cited by Institution
2 students, 21,
or 7%, were not owned by the institution,
93% were owned. Of the 362 references
cited by Institution 3 students, only 11, or 3%, were not locally owned,
97% were.
To arrive at some explanation of student reliance on local collections
dissertation citations were scored for scholarliness, currency, and appropriateness
of
format. The criterion of scholarliness was scored based on journal prestige
within the
discipline and the field, presence or absence of peer review, and consideration
of
empirical, research-based studies rather than program descriptions. Citations were
also
rated on currency, or their timeliness of publication. The date of publication was
considered in context of type of material and usage in the literature review, and
the
raters recognized when currency was not an issue. Appropriateness, or
fit of the
material type to the topic being developed, was considered in relation to maturity
of the
field. Scores on the three criteria were averaged to arrive at an overall quality
rating.
Across all coded citations, the mean statistic for scholarliness was 2.70 (SD =
.80), skewness was .164 (SE = .057), and kurtosis was -.752 (SE = .114). Statistics
for
the remaining criteria include: currency (M = 2.63, SD = .56, skewness = -1.243, and
kurtosis = .560), appropriateness (M = 2.68, SD = .56, skewness = -1.534, and kurtosis
= 1.383), and quality score (M = 2.67, SD = .45,
skewness = -.398, and kurtosis = -
.478). Descriptive statistics for each criterion and by institution are shown in Tables 7
through 10. Scores were also submitted to the Lilliefors Significance Correction of the
Kolmogorov-Smirnov test of normality. Normality statistics are reported in Table 11,
and boxplots, see Figures 1 through 4, offer a graphic representation of the
distributions.
A Kruskal-Wallis test was conducted comparing the scores on coded citations
across institutions. A statistically significant
result was found for scholarliness (H(2) =
107.11, p < .01), indicating that the institutions differed from each other. Institution 2
averaged a placement of 774.37, while Institution 1 averaged a placement of 978.70
and Institution 3 averaged 1038.20. Currency also differed significantly (H(2) = 43.11,
p < .01) across institutions. Institution 2 averaged a
rank of 847.61 while Institution 1
averaged 918.41 and Institution 3 999.74. A statistically significant result was found
for appropriateness scores (H(2) ,--- 57.70, p < .01) when compared across institutions.
Institution 2, with an average rank of 829.82, was lower than Institution 3, at 975.81
and Institution 1, at 986.95. Quality scores were likewise significantly different (H(2)
= 150.32. p < .01). Institution 2 averaged 739.72
while Institution 1 averaged 988.36
and Institution 3 1068.03.
A one-way ANOVA was also calculated comparing each of the criteria across
institutions. For scholarliness scores, a statistically significant difference was found
(F(2,1839) = 52.36, p < .01). Tukey's HSD was calculated to determine the nature of
9
Characteristics of Education Doctoral Dissertation
References, p. 8
the differences among institutions. This analysis
revealed that Institution 1 (M = 2.79,
SD = .75) and Institution 3 (M = 2.88, SD =
.82) dissertation citations were scored
higher than Institution 2 (M = 2.47, SD = .74)
dissertation citations on scholarliness.
Scholarliness scores were not significantly different
for either of the other two groups.
A statistically significant difference among
institutions was also found with
currency scores (F(2,1839) =
25.60, p < .01). Post hoc analysis revealed each
institution varied significantly from the other,
Institution 1 (M = 2.64, SD = .54),
Institution 2 (M = 2.53, SD = .63), and Institution 3
(M = 2.74, SD = .47).
Appropriateness scores were likewise significantly
different (F(2,1839) = 37.83, p <
.01) and Tukey's HSD revealed that Institution 1
(M = 2.77, SD = .46) and Institution 3
(M = 2.76, SD = .46) scored higher on appropriateness
than Institution 2 (M = 2.54, SD
= .66). Appropriateness scores were
not significantly different from either of the
remaining groups.
A one-way ANOVA was also computed for the
quality rating, which was an
average of the criteria reported above.
Analysis indicated a statistically significant
difference was found for overall quality of coded references among
institutions
(F(2,1839) = 78.70, p < .01). Tukey's HSD revealed Institution 2
(M = 2.51, SD = .45)
varied significantly from Institution 1 (M= 2.73, SD = .41) and
Institution 3 (M = 2.79,
SD = .43), but quality scores did not significantly vary from either
of the other
institutions. Mean scores by institution for each criterion are displayed on
Figures 5
through 8.
Analysis results were considered in conjunction with institutional
characteristics. The less well-established school, Institution 2, systematically
received
lower scores across all criteria, which appeared to offer support to the U. S.
News &
World Report schools of education (2000) rankings. Results may also be
explained by
the heavy reliance of students from Institution 2 on sources other than
scholarly
journals and books. "Other" items, including ERIC documents, doctoral dissertations,
and abstracts of dissertations, along with magazines and Web sites, accounted
for over
one-third of coded references from Institution 2. The literature is explicit in its
emphasis on primary, scholarly resources (Babbie, 1998; Creswell, 1994; Fraenkel &
Wallen, 1990; among others).
Results were also regarded alongside published standards for quality literature
reviews. The purpose of the review is to provide a framework for establishing
the
importance of the study and for relating the results to other findings (Creswell, 1994),
and technical advice for authors often includes guidelines for performing the
review of
literature. Included in these suggestions are the criteria of relevance and completeness,
and synthesis and analysis. Specific to references, Creswell (1994) considers what
types of literature might be reviewed and in what priority. Foremost are
journal articles
from respected national journals, then books that offer research summaries of the
scholarly literature. With the admonition that one needs to be highly selective as
quality varies considerably, other items to contemplate might include recent
conference
papers from major national conferences and
dissertations.
10
Characteristics of Education Doctoral Dissertation References, p. 9
To investigate further, the five highest scored dissertations were
examined in
more depth and were found to be
frequently dated, partial, non-scholarly, or heavily
reliant upon others' reviews or textbooks. This evidence suggests
that dissertation
citations were not consistently commented upon by reviewing
faculty and/or standards
for acceptable bibliographies were of no or little importance.
Variation among
institutions may be illustrative of differences in institutional
characteristics, but may
also be attributed to the faculty's expertise and assumed
responsibility in
communicating scholarship expectations.
Importance of the Study
Importance of the study can be discussed on two fronts; doctoral
student
performance as demonstrated by quality of dissertation references, and use of
citation
analysis for collection decisions. The assumption that doctoral students are on
cutting
edge of current research was not supported by this study, and professors
of educational
research should be concerned that students are not adequately mining the
academic
literature in the field. Certainly, it can be argued that citation behavior may
have little
relationship to the quality of the research performed by the student; however, it is also
reasonable to expect that as part of their professional preparation students be fully
conversant in accessing and evaluating the scholarly literature, and able to
demonstrate
this via dissertation references.
Professors, if they wish to see an improvement in the resources cited by
students, will have to present more clearly defined expectations of the cited literature
and be willing to offer feedback regarding the quality of cited references. Tuckman
(1990) suggests that existing strategies of manuscript evaluation are clearly inadequate
and, due to the lack of consensus on research standards in the field, offers a framework
from which to evaluate the literature review. At minimum, this study suggests further
local investigation as to whether graduate students satisfactorily review all resources,
and the need for increased faculty engagement as expressed by higher standards, more
intensive instruction, and attention to the literature review process.
Grant and Graue (1999) explored the concept of what constitutes evidence in
reviews and called for more rigorous standards to determine credible research.
Cognizant of the fact that the density of work available on a given topic varies, they
recommend the lack of acceptable research be discussed in the review. Of the reviewed
dissertations, only two included the criteria used to identify source materials. This
methodology is suggested to inform the reader as to the extent of the information
sought and is an indication of the amount and quality of literature available on the
topic. An example of the methodology used that might be included in the review of
literature chapter follows:
The published literature was searched using ERIC, Education Abstracts,
Dissertation Abstracts, and PsycINFO databases. Higher Education Abstracts
was searched in print. In addition, the bibliographies of all acceptable
studies
11
Characteristics of Education Doctoral Dissertation References, p. 10
and review articles from the past two years were searched for potentially
relevant citations.
English language published literature from 1985 through current year (2000)
was sought, utilizing the following search terms:
Learning communities; collaboration OR cooperation; literacy
Citation analysis studies are often used as a basis for collection management
decisions, but there is a question of validity as to what questions these studies can
answer. Library collections at each of the
institutions examined held a large majority of
the materials cited by the doctoral students. While previous research has
assumed this
indicates an adequate collection, the results from this study suggest that doctoral
students simply do not possess sufficient knowledge of information resources, expertise
in mining the literature of the field, or the ability to consistently discriminate between
popular and scholarly resources to create quality bibliographies.
Results of this study support Kuyper-Rushing's (1999) findings that analysis
from a single institution could result in a skewed list of journals. This study likewise
found a journal list derived from dissertation reference analysis from a single institution
varied significantly from a list generated through analysis of a larger institutional base.
Students do not appear to seek sources not locally owned; and thus, it may be inferred
single institution journal lists can be used to reflect local use, but do not necessarily
provide information on which journals should be added to the collection. Citation
analysis may be valuable for serials cancellation projects, but using single institution
analysis to indicate collection adequacy should proceed cautiously.
Ultimately, whether due to graduation or attrition the doctoral student
population is by nature transient, and basing collection decisions on their research
interests and information searching prowess should not be the sole means of
determining a core journal collection. Only after the quality of dissertation references
is established and core lists are created by comparison to external institutions can a
journal list be considered as one tool for building the research collection. To arrive at a
more robust indication of collection use and needs, it is
suggested citation analysis be
used in conjunction with other methods, such as journal impact ratings and faculty
publication citation analysis.
12
Characteristics of Education Doctoral Dissertation References, p.
11
References
Babbie, E. (1998). The practice of social research
(8th ed.). Belmont, CA: Wadsworth
Publishing.
Barry, C. A. (1997). Information skills for an
electronic world: Training doctoral
research students. Journal of Information Science, 23(3),
225-238.
Buttlar, L. (1999). Information sources in library and
information science doctoral
research. Library & Information Science Research, 21(2),
227-245.
Creswell, J. W. (1994). Research design: Qualitative and
quantitative approaches.
Thousand Oaks, CA: Sage Publications.
Davis, P. M. (2002). The effect of the Web on undergraduate
citation behavior: A 2000
update. College & Research Libraries, 63(1), 53-60.
Davis P. M. & Cohen, S. A. (2001). The effect of the Web on
undergraduate citation
behavior, 1996-1999. Journal of the American Society for Information
Science
and Technology, 52(4), 309-314.
Diadoto, V. (1994). Dictionary of bibliometrics. Binghampton, NY: Haworth Press.
Fraenkel, J. R. & Wallen, N. E. (1990). How to design and evaluate research in
education (3rd ed.). New York: McGraw-Hill.
Gooden, A. M. (2001). Citation analysis of chemistry doctoral dissertations: An Ohio
State University case study. Issues in Science and Technology Librarianship,
32. Retrieved January 22, 2002, from http://www.istl.org/ist1/01-
fal I/refereed. html
Grant, C. A. & Graue, E. (1999). (Re)Viewing a review: A case history
of the
"Review of Educational Research." Review of Educational Research, 69(4),
384-396.
Hall, B. W., Ward, A. W. & Corner, C. B. (1988). Published educational research: An
empirical study of its quality. Journal of Educational Research, 81(3), 182-189.
Hernon, P. (1994). Serious stuff to ponder. Library & Information Science Research,
16, 271-278.
Herring, S. D. (2002). Use of electronic resources in scholarly electronic journals: A
citation analysis. College & Research Libraries, 63(4), 334-340.
Kohl, D. F. & Wilson, L. A. (1986). Effectiveness of course-integrated bibliographic
instruction in improving coursework. RQ, 26, 206-211.
Characteristics of Education Doctoral Dissertation References, p. 13
TABLE 1
Dissertation Information by Institution
Average Number of
Average Number of
Average Number
Institution
Total Citations
Citations Coded
of Pages
1
83.60 (SD-31.40)
42.90 (SD-16.55)
199.40 (SD=74.32)
2
90.40 (SD=30.63)
71.80 (SD=34.89)
119.60 (SD-24.38)
3
89.10 (SD=38.21)
69.50 (SD=35.25)
119.30 (SD=44.30)
TOTAL
87.70 (SD=32.54)
61.40 (SD=32.01)
146.10 (SD-63.06)
TABLE 2
Material Type by Institution
Journal
Mono-
Maga-
Web
Institution
Articles
graphs
"Other"
zines
Sites
Total
1
Count
188
188
44
8
1
429
% within
Institution
43.8%
43.8% 10.3%
1.9%
.2% 100.0%
2
Count
284
187
225
14 8
718
% within
Institution
39.6%
26.0%
31.3%
1.9%
1.1% 100.0%
3
Count
357
249
68 6
15
695
% within
Institution
51.4%
35.8% 9.8%
.9%
2.2%
100.0%
TOTAL
Count
829
624
337
28
24
1842
% of
Total
45.0%
33.9% 18.3%
1.5%
1.3%
100.0%
15
Characteristics of Education Doctoral Dissertation
References, p. 14
TABLE 3
Core Journal Title List with Citation Count
Core Journal Titles
Citation
Count
Journal of learning disabilities
36
Exceptional children
32
Journal of educational psychology
25
Phi Delta Kappan
20
Remedial and special education
20
Psychology in the schools
19
American educational research journal
17
Child development
13
Learning disabilities research and practice
13
Learning disability quarterly
13
Review of educational research
13
Evaluation and program planning
12
Journal of special education
12
Reading research quarterly
12
Educational leadership
11
Journal of educational research
11
Teaching exceptional children
11
TABLE 4
Journal Title List with 5 or More Citations, Institution 1
Journal Titles
Citation
Count
Journal of learning disabilities
13
Exceptional children
8
Psychology in the schools
7
School counselor
7
School psychology review
7
NASSP bulletin
6
Phi Delta Kappan
6
Journal of moral education
5
16
Characteristics of Education Doctoral Dissertation References, p. 15
TABLE 5
Journal Title List with 5 or More Citations, Institution 2
Journal Titles
Citation
Count
Exceptional children
19
Evaluation and program planning
12
Journal of educational psychology
12
Remedial and special education
11
Journal of learning disabilities
10
Chronicle of higher education
7
Journal of research on computing in education
7
Journal of special education
7
Phi Delta Kappan
7
Journal for research in mathematics education
6
Journal of educational research
6
Review of educational research
6
Teaching exceptional children
6
Educational technology
5
Journal of counseling psychology
5
Journal of reading
5
Learning disability quarterly
5
17
Characteristics of Education Doctoral Dissertation References, p. 16
TABLE 6
Journal Title List with 5 or More Citations, Institution 3
Journal Titles
Citation
Count
American educational research journal
14
Journal of learning disabilities
13
Child development
12
Journal of educational psychology
12
Learning disabilities research and practice
11
Early childhood research quarterly
10
Psychology in the schools
10
Reading research quarterly
10
Phi Delta Kappan
7
Topics in early childhood special education
7
Childhood education
6
Journal of early Intervention
6
Learning disability quarterly
6
Psychological bulletin
6
Remedial and special education
6
Review of educational research
6
Young children
6
Educational leadership
5
Elementary school journal
5
Exceptional children
5
Journal of educational research
5
Journal of teacher education
5
Characteristics of Education Doctoral Dissertation References, p. 17
TABLE 7
Scholarliness, Selected Descriptive Statistics, by Institution
Institution
Statistic
S.E.
1
Mean
2.790
3.610
Median
3.000
St. Dev.
.750
Skewness
.099
.118
Kurtosis
-.722
.235
2
Mean
2.470
2.780
Median
2.000
St. Dev.
.740
Skewness
.544
.091
Kurtosis
-.240
.182
3
Mean
2.880
3.100
Median
3.000
St. Dev.
.820
Skewness
-.176
.093
Kurtosis
-.728
.185
TABLE 8
Currency, Selected Descriptive Statistics, by Institution
Institution
Mean
Median
St. Dev.
Skewness
Kurtosis
Statistic
S.E.
1
2.640
3.000
.540
-1.137
.285
2.610
.118
.235
2
Mean
2.530
2.360
Median
3.000
St. Dev.
.630
Skewness
-.994
.091
Kurtosis
-.092
.182
3
Mean
2.740
1.800
Median
3.000
St. Dev.
.47
Skewness
-1.523
.093
Kurtosis
1.293
.185
Characteristics of Education Doctoral Dissertation References, p.
18
TABLE 9
Appropriateness, Selected Descriptive Statistics, by Institution
Institution
Mean
Median
St. Dev.
Skewness
Kurtosis
Statistic
S.E.
1
2.770
3.000
.460
-1.821
2.449
2.200
.118
.235
2
Mean
2.540
2.460
Median
3.000
St. Dev.
.660
Skewness
-1.127
.091
Kurtosis
.058
.182
3
Mean
2.760
1.740
Median
3.000
St. Dev.
.460
Skewness
-1.678
.093
Kurtosis
1.835
.185
TABLE 10
Quality Score, Selected Descriptive Statistics, by Institution
Institution
Mean
Median
St. Dev.
Skewness
Kurtosis
Statistic
S.E.
1
2.730
2.660
.411
-.502
-.236
1.980
.118
.235
2
Mean
2.510
1.693
Median
2.666
St. Dev.
.453
Skewness
-.060
.091
Kurtosis
-.545
.182
3
Mean
2.794
1.638
Median
3.000
St. Dev.
.431
Skewness
-.722
.093
Kurtosis
.083
.185
' 0
4.
Characteristics of Education Doctoral Dissertation References, p. 19
TABLE 11
Summary Statistics, Tests of Normality
Kolmogorov-Smirnov (a)
Criteria
Institution
Statistic
df
Scholarliness
1
.241
429
.000
2
.326
718
.000
3
.229
695
.000
Currency
1
.416
429 .000
2
.375
718
.000
3
.463
695
.000
Appropriateness
1
.480
429
.000
2
.391
718
.000
3
.475
695
.000
Quality Score
1
.206
429
.000
2
.146
718
.000
3
.223
695
.000
a Lilliefors Significance Correction
21
4.5
4.0 ,
3.5
3.0 -
2.5
2.0
Characteristics of Education Doctoral Dissertation Kererences, p.
hu
FIGURE 1
Boxplots of Scholarliness, by Institution
1.5
1.0
.5
N.
Institution
429
718
695
1
2
3
FIGURE 2
Boxplots of Currency, by Institution
3.5
3.0
2.5 -
2.0
1.5
1.0
.5
N.
Institution
429
1
718
2
695
3
22
3.5
3.0 -
2.5
2.0
1.5
1.0
Characteristics of Education Doctoral Dissertation References, p. 21
FIGURE 3
Boxplots of Appropriateness, by Institution
*ON
N=
Institution
429
1
718
2
FIGURE 4
Boxplots of Quality Score, by Institution
a)
695
3
N
Institution
429
718
695
1
2 3
23
Characteristics of Education Doctoral Dissertation References, p. 22
FIGURE 5
Mean of Scholarliness, by Institution
3.0
2.9
2.8
2.7
2.6
2.5
2.4
1
Institution
FIGURE 6
Mean of Currency, by Institution
2
3
Institution
24
Characteristics of Education Doctoral Dissertation References, p. 23
FIGURE 7
Mean of Appropriateness, by Institution
Institution
FIGURE 8
Mean of Quality Score, by Institution
2.9
2.8
.
2.7
2.6 .
2.5
2.4
.
Institution
2
25
3
TATES 01
U.S. Department of Education
Office of Educational Research and Improvement (OERI)
National Library of Education (NLE)
Educational Resources Information Center (ERIC)
REPRODUCTION RELEASE
DOCUMENT IDENTIFICATION:
(Specific Document)
r/.
n1
ERIC
Wilma:uses Wyman tato
Title: Characteristics of Education Doctoral Dissertation References: An
Inter-Institutional Analysis of Review of Literature Citations
Author(s): Beile,
Penny M.; Boote, David N.; Killingsworth, Elizabeth K.
Corporate Source:
University of Central Florida
Publication Date:
04/25/2003
II.
REPRODUCTION RELEASE:
In order to disseminate as widely as possible timely and significant materials of interest to the educational community, documents
announced in the monthly abstract joumal of the ERIC system, Resources in Education (RIE), are usually made available to users in microfiche,
reproduced paper copy, and electronic media, and sold through the ERIC Document Reproduction Service (EDRS). Credit is given to the
source of each document, and, if reproduction release is granted, one of the following notices is affixed to the document.
If permission is granted to reproduce and disseminate the identified document, please CHECK ONE of the following three options and sign
at the bottom of the page.
The sample sticker shown below will be
affixed to all Level 1 documents
PERMISSION TO REPRODUCE AND
DISSEMINATE THIS MATERIAL HAS
BEEN GRANTED BY
TO THE E UCATIONAL RESOURCES
INFORMATION CENTER (ERIC)
Level 1
Check here for Level 1 release, permitting
reproduction and dissemination in microfiche or
other ERIC archival media (e.g., electronic) and
paper copy.
The sample sticker shown below will be
affixed to all Level 2A documents
2A
PERMISSION TO REPRODUCE AND
DISSEMINATE THIS MATERIAL IN
MICROFICHE, AND IN ELECTRONIC MEDIA
FOR ERIC COLLECTION SUBSCRIBERS ONLY,
HAS BEEN GRANTED BY
TO THE E UCATIONAL RESOURCES
INFORMATION CENTER (ERIC)
Level 2A
[01
Check here for Level 2A release, permitting reproduction
and dissemination in microfiche and in electronic media for
ERIC archival collection subscribers only
The sample sticker shown below will be
affixed to all Level 2B documents
PERMISSION TO REPRODUCE AND
DISSEMINATE THIS MATERIAL IN
MICROFICHE ONLY HAS BEEN GRANTED BY
2B
SSC
TO THE EDUCATIONAL RESOURCES
INFORMATION CENTER (ERIC)
Level 2B
Check here for Level 2B release, permitting reproduction
and dissemination in microfiche only
Documents will be processed as indicated provided reproduction quality permits.
If permission to reproduce is granted, but no box is checked, documents will be processed at Level 1.
I hereby grant to the Educational Resources Information Center (ERIC) nonexclusive permission to reproduce and disseminate this document
as indicated above. Reproduction from the ERIC microfiche or electronic media by persons other than ERIC employees and its system
contractors requires permission from the copyright holder. Exception is made for non-proft reproduction by libraries and other service agencies
to satisfy information,needs of educators in response to discrete inquiries.
Signature:
Printed Name/Position/Title:
.....,.-
,5-
Penny Beile/Associate Librarian
Organizatiolddress:
Uni ersity of Central Florida
Telephone: (407) 823-548E FAX: (407) 823-5488
PO Box 162666, Orlando, FL 32816-2666
E-134M4mail.uctedu
Date: 04/25/2003