references of educational research

• Ask a Librarian
• Education Reference Sources
• Citation Searching
• Dissertation Writing
• Publication Style Guides
• Tutorials for Searching Education Topics
• ERIC Documents and Database
• Other Relevant Databases
• Online Journals
• Dissertations and Conference Proceedings
• K-12 Historic Textbooks
• Open Access
• Tests & Measurements
• Video & Media
• Education Awards
• K-12 Outreach
• Lesson Plans
• Standards - Curriculum for US states
• Teaching Degrees
• Useful Library Guides

Clinical Assistant Professor

Education Encyclopedias

• World Education Encyclopedia Comparative, in-depth essays on the educational systems of 233 countries and/or territories. World Education Also contains custom-made graphs and statistical tables, as well as regional maps and an extensive index.
• Encyclopedia of Education Articles on institutions, people, processes, roles, and philosophies in educational practice in the USA and the world. Features biographies, profiles of universities and of organizations; and an appendix of full-text documents, including legislation, international treaties, and testing methods.
• International Encyclopedia of Education Unequalled in its combination of authoritative scholarship and comprehensive coverage, International Encyclopedia of Education, Third Edition succeeds two highly successful previous editions (1985, 1994) in aiming to encapsulate research in this vibrant field for the twenty-first century reader. Under development for five years, this work encompasses over 1,000 articles across 24 individual areas of coverage and is expected to become the dominant resource in the field.

Other Education Encyclopedias

• Cambridge Encyclopedia of Child Development The Cambridge Encyclopedia of Child Development remains the most authoritative and accessible account of all aspects of child development. Written by an international team of experts, its comprehensive coverage includes everything from prenatal development to adolescence, pediatrics, theories and research methods, physical development, social and emotional development, perceptual and cognitive development, language development, psychopathology, and parenting.
• Encyclopedia of Cross-Cultural School Psychology The Encyclopedia of Cross-Cultural School Psychology (ECCSP) is comprehensive and reader-friendly, with approximately 400 entries written by leading researchers, educators, and practitioners in the fields of school psychology and education. ECCSP provides an easily accessible A-to-Z reference in one concise volume across these six key cross-cultural competency areas: Legal and Ethical Issues; School Culture, Educational Policy, and Institutional Advocacy; Psychoeducational Assessment and Related Issues; Academic, Therapeutic, and Consultative Intervention; Working with Interpreters; and Research.
• Encyclopedia of Distance Learning Encompasses the latest concepts, trends, issues, and technologies in the field of distance learning, providing an audience of practitioners, researchers, educators, and students with a critical mass of knowledge on an emerging and significant educational field of study. Containing over 100 research articles by internationally-renowned professionals, this must-have resource contributes the latest findings and practices in topic areas such as computer-based learning, teaching methodologies, and distance learning programs.
• Encyclopedia of Giftedness, Creativity, and Talent Covers all major facets of the field, including achievement motivation, artistic ability, creative personality, emotional intelligence, gender differences, genius, intelligence testing, learning styles, minority underrepresentation, multiple intelligences, musical ability, prodigies, scientists, self actualization, thinking skills, and more.
• Encyclopedia of Information Technology Curriculum Integration A comprehensive resource of concepts, methodologies, models, architectures, applications, enabling technologies, and best practices for integrating technology into the curriculum at all levels of education. Compiling 154 articles from over 125 of the world's leading experts on information technology, this authoritative reference strives to supply innovative research aimed at improving academic achievement, teaching and learning, and the application of technology in schools and training environments.
• Encyclopedia of Language and Education Includes research and scholarly content essential to the field of language teaching and learning in the age of globalization. In the selection of topics and contributors, the Encyclopedia reflects the depth of disciplinary knowledge, breadth of interdisciplinary perspective, and diversity of sociogeographic experience in the language and education field.
• Encyclopedia of Special Education Addresses issues of importance ranging from theory to practice and is a critical reference for researchers as well as those working in the special education field.
• Gender and Education : An Encyclopedia In this two volume set, educators explore the intersection of gender and education. Their entries deal with educational theories, research, curricula, practices, personnel, and policies, but also with variations in the gendering of education across historical and cultural contexts.
• Philosophy of Education: An Encyclopedia The encyclopedia is designed to show the diversity of topics that contribute to the study of the philosophy of education.
• TESOL Encyclopedia of English Language Teaching The TESOL Encyclopedia of English Language Teaching publishes new content online twice a year (summer and winter) to ensure that it remains the most up-to-date reference in the field of English language teaching available for students, researchers, academics and professionals.
• Education Section of CREDO

CREDO is an online reference library that provides advanced searching and full-text of hundreds of dictionaries and encyclopedias in the humanities, social sciences, and sciences.

In Education area, it includes titles such as:

• The Cambridge Guide to Teaching English to Speakers of Other Languages
• The Development Dictionary
• Dictionary of Youth Justice
• Encyclopedia of Special Education
• Gender and Education: An Encyclopedia
• Handbook of Research on Teaching the English Language Arts
• Handbook of Research on the Education of Young Children
• International Handbook of Giftedness and Talent
• Key Concepts in Education
• Philosophy of Education: An Encyclopedia
• SAGE Key Concepts series: Key Concepts in Learning Disabilities
• Springer International Handbooks of Education: International Handbook of Educational Evaluation
• Springer International Handbooks of Education: International Handbook of School Effectiveness and Improvement
• << Previous: Research Guidance
• Next: Citation Searching >>
• Last Updated: Jun 4, 2024 2:30 PM
• URL: https://guides.lib.purdue.edu/education

Research in Education

• Lit Review - Best Sources
• Books and Dissertations
• Interlibrary Loan
• How-To for Google Scholar
• How-To for ERIC (EBSCO)
• How-To for ERIC (Free)
• How-To for APA PsycInfo
• How-To for OneSearch
• Is it Peer Reviewed?
• Finding the DOI

APA Citations

• Purdue OWL Citing online journal articles and books.
• APA Style Blog Citation advice from APA itself.
• How to Use the New DOI Format in APA Style Sage advice on DOI from APA

Annotated Bibliographies

• Bethel University Library Example of an annotated bibliography in APA style.
• Cornell University One APA and one MLA example.
• Purdue OWL General advice for annotated bibliographies.
• Writing Center at UNC-Chapel Hill Multiple examples. Note - none of the examples include online research articles.

Contact Your Librarian

APA 7th - from APA (American Psychological Association)

• APA 7th Quick Guide
• APA Style Reference Examples From APA : A guide to creating references for a wide-variety of formats.
• APA In-Text Citations From APA : A guide to creating in-text citations for a wide-variety of formats. Advice on paraphrasing, quotations, and plagiarism.
• APA Paper Format From APA: A guide to headings, margins, paragraph indentation, spacing, and title page format,
• APA Sample Papers From APA: A sample professional paper and a sample student paper.

APA 7th - Purdue OWL

• Citing Journal Articles (Purdue OWL)
• Headings - APA 7th (Purdue OWL)
• In-Text Citations - APA 7th - Basics (Purdue OWL)
• In-Text Citations - APA 7th - More than one author (Purdue OWL)
• Sample Paper - APA 7th (Purdue OWL) Both a sample professional paper and a sample student paper are available.
• Tables and Figures - APA 7th (Purdue OWL)

Examples of APA Citations for Journal Articles

Citing journal articles in APA 7th with a Direct Object Identifier (DOI).

(Reminder - find the DOI with CrossRef ).

Alkharusi, H. (2017). Predicting students’ academic achievement: Contributions of perceptions of classroom assessment tasks and motivated learning strategies. Electronic Journal of Research in Education Psychology 14 (40) 515-533. https://doi.org/10.14204/ejrep.40.15177

Kjeldsen, T. H., & Blomhøj, M. (2012). Beyond motivation: History as a method for learning meta-discursive rules in mathematics. Educational Studies in Mathematics , 80 (3), 327-349. https://doi.org/10.1007/s10649-011-9352-z

Weidinger, A. F., Steinmayr, R., & Spinath, B. (2017). Math grades and intrinsic motivation in elementary school: A longitudinal investigation of their association. British Journal of Educational Psychology , 87 (2), 187-204. https://doi.org/10.1111/bjep.12143

For articles without a Direct Object Identifier ( CrossRef does not find a DOI for this article) simply use the web address (URL) associated with the article.

Pourdavood, R. G., Grob, S., Clark, J., & Orr, H. (1999). Discourse and professional growth: Processes, relationships, dilemmas, and hope. School Community Journal , 9 (1), 33-48. http://www.adi.org/journal/ss99/PourdavoodGrobClarkOrrSpring1999.pdf

Examples of APA 7th Citations for Books and Book Chapters

Citing Books and Chapters of Books

APA 7th guidelines:

Treat the authors, date, and title as before (with the title in italics).

APA 7th requires an edition number after the title e.g. (4th ed) but this is not in italics.

APA 7th no longer requires a “place of publication” - just the name of the publisher.

APA 7th - For ebooks only use a DOI if it exists. Only use a URL if:

• the book does not have a DOI 
• the URL is NOT from an academic research database . 

For an entire book in print cite as follows:

Carr, M. (1996). Motivation in mathematics . Hampton Press.

For a chapter from a book in print cite as follows:

Dorfler, W. (1999). Mathematics provides tools for thinking and communicating. In C. Hoyles, C. Morgan, & G. Woodhouse (Eds.), Rethinking the mathematics curriculum (pp. 75-86). Falmer Press. 

For an entire ebook (that does not have a DOI) cite as follows:

Hall, N. C. & Goetz, T. (2013). Emotion, motivation, and self-regulation: A handbook for teachers . Emerald Group Publishing. https://northern-iowau-primo.hosted.exlibrisgroup.com/permalink/f/1a1g5a8/01NRTHIOW_ALMA51128858790002841

For a chapter from an ebook that has a DOI cite as follows:

Harackiewicz, J., Tibbetts, Y., Canning, E., & Hyde, J. (2014). Harnessing values to promote motivation in education. In S. Karabenick & T. Urdan (Eds.), Motivational interventions (pp. 71-105). Emerald Group Publishing. https://doi.org/10.1108/s0749-742320140000018002

• << Previous: Finding the DOI
• Last Updated: Aug 22, 2023 3:29 PM
• URL: https://guides.lib.uni.edu/education-research

• University of Michigan Library
• Research Guides
• Reference Sources
• Getting started
• Databases/Finding Articles
• Finding Books
• Research Data
• Useful Library Services
• Open Access Resources

Selected Online Reference Sources

Breadcrumbs Section. Click here to navigate to respective pages.

Foundations of Education Research

DOI link for Foundations of Education Research

Get Citation

Now in its second edition, Foundations of Education Research defines, discusses, and offers applications for the central components of educational research, providing both novice and experienced researchers with a common ground from which to work. Fully updated throughout, the second edition adds a glossary of terms, additional examples, and includes a discussion of similarities and differences in education research. Eight concise, accessible chapters cover conceptual framework, epistemology, paradigm, theory, theoretical framework, and methodology/method. This unique primer demystifies jargon and makes the theoretical components of research accessible, giving students the tools they need to understand existing education research literature and to produce theoretically-grounded work of their own.

Each chapter begins with perspectives from both novice and experienced researchers, whose guiding questions assist researchers engaging with theory for the first time and those looking to improve their understanding of the fundamentals. Practice exercises, examples, and suggested reading lists at the end of each chapter offer students resources they can apply to their own research and thinking in concrete ways. A perfect accompaniment to standard research courses, this book is designed to help students achieve a deeper understanding of what is expected of them and ideas about how to achieve it.

TABLE OF CONTENTS

Chapter 1 | 5  pages, introduction, chapter 2 | 11  pages, foundations of research: conceptual framework, chapter 3 | 13  pages, influences on the research and the researcher: episte-what, chapter 4 | 15  pages, research paradigms, chapter 5 | 11  pages, research and theory: just a hunch or something more, chapter 6 | 15  pages, theoretical frameworks: you can’t have a how without a why, chapter 7 | 15  pages, research methodologies and methods: ingredients for research success, chapter 8 | 8  pages, myths and misconceptions about research.

• Privacy Policy
• Terms & Conditions
• Cookie Policy
• Taylor & Francis Online
• Taylor & Francis Group
• Students/Researchers
• Librarians/Institutions

Connect with us

Registered in England & Wales No. 3099067 5 Howick Place | London | SW1P 1WG © 2024 Informa UK Limited

What Is in a Reference? Theoretically Understanding the Uses of Evidence in Education Policy

• Open Access
• First Online: 02 April 2022

Cite this chapter

You have full access to this open access chapter

• Gita Steiner-Khamsi 3  

3415 Accesses

The chapter deals with the reference—the unit of analysis for our bibliometric analyses—and examines what it stands for in the policy process. We found Paul Cairney’s ( The Politics of Evidence-Based Policymaking . Palgrave Pivot, 2015) definition useful: “‘[e]vidence’ is assertion backed by information.” In concert with Cairney’s definition, we treat references as a construct or an aggregate of several pieces of information (authorship, year of publication, topic or theme, etc.) that helps position the author in a larger semantic space. We have used all these constitutive elements as epistemological cues for understanding not only whose texts or whose knowledge the authors have selected to substantiate their points with, but also whose knowledge they cite as sources of expertise in order to reduce uncertainty, enhance credibility, or generate legitimacy about the validity of their own claims or assertions. In an era in which we have a surplus of information as well as a surplus of evidence, this is no small enterprise.

You have full access to this open access chapter,  Download chapter PDF

Similar content being viewed by others

Capturing the ‘Evidence’ and ‘What Works’ Agenda in Education: A Truth Regime and the Art of Manoeuvring Floating Signifiers

Finding the Poster Child: Reference Patterns in OECD Country Reports

The Globalized Expert: On the Dissemination and Authorization of Evidence-Based Education

• Education policy

Arguably, the main unit of analysis for all bibliometric analyses is the reference. It may be listed conveniently at the end of a document in a separate bibliographical section, in a footnote, in an endnote or, more inconveniently, only vaguely alluded to in the main text. In either case, the reference communicates or conveys “something” to the reader. One question arises: What exactly does it communicate, what is embedded in a reference, or, to place the question into the context of our study, how have we interpreted the reference in the larger corpus of policy documents that we collected in Denmark, Iceland, Finland, Norway, and Sweden in an attempt to understand the policy process?

References have several constitutive elements: an indication of authorship, year of publication, topic or theme, location of publisher, and type of publisher. The authorship may be further differentiated by gender, nationality, institutional affiliation, and, if several coauthors are involved, the network structure within the group of authors. All these constitutive elements are essential in a bibliometric analysis because they are utilized as epistemological cues for understanding not only whose texts or whose knowledge the authors have selected to substantiate their points, but also whose knowledge they cite as sources of expertise to reduce uncertainty or generate legitimacy about the validity of their own claims or assertions.

According to political scientist Paul Cairney ( 2015 ), “‘[e]vidence’ is assertion backed by information.” In our bibliometric network analysis, we treat references as a construct or aggregate of several pieces of information (authorship, year of publication, topic or theme, etc.), helping position the author in a larger semantic space. Referencing sources in a policy document—whether it is a Green Paper (in the Nordic context: prepared by government-appointed expert panels) or a White Paper (issued by the government)—is fundamentally different from the references in, for example, an encyclopedia, where the expectation is that the researcher at least pretends to identify the universe of relevant research literature on the topic with which they can situate their own framework. Policy documents are much more evaluative and controversial in nature; therefore, they tend to openly take a stance for or against existing assertions. Unsurprisingly, a bibliometric analysis of policy documents will often surface clusters of like-minded authors/documents and set them apart from those holding viewpoints that are seen as different or distant.

References from the Perspective of Sociological Systems Theory

In the Nordic POLNET (Policy Knowledge and Lesson Drawing in an Era of International Comparison) study, we interpret a reference functionally, that is, in its larger context of evidence-based policy planning. We ask the following: What does a reference stand for—or rather do—in a policy document? From the perspective of sociological systems theory, references are meant to reduce uncertainty for the reader. They do this by making transparent the positionality of the author in the larger, discursive policy space and by documenting the credibility of the sources used for the assertions the author is making. As Jenny Ozga ( 2019 ) has astutely pointed out, the discursive space in policy documents is by default a political space in which authors position themselves in terms of political orientation and alliances.

To reiterate, references help validate or provide legitimacy to the evidence that the author (e.g., the government-appointed expert commissions or the government) has presented in the document. Thus, if “evidence is assertion backed by information” (Cairney, 2015 ), then a reference is validation of evidence . Said differently, references are used to provide authoritative status to the evidence presented in policy documents. Naturally, a host of questions surface with this particular conceptualization of references: Which texts are influential, that is, referenced frequently or referenced by two or more different knowledge networks? Are international references, that is, texts published outside the Nordic region, more influential than national or regional references or vice versa? Does the institutional affiliation (government, academe, “institute sector,” private think-tank, civil society organization) of the author matter? Finally, the cross-national dimension enables us to examine the varied legitimization or authorization strategies in the five countries in depth and, by means of comparison, identify nation-specific patterns in the policy process.

Eyal ( 2019 ) explains Luhmann’s conceptualization of authority, validity, and legitimacy of expert knowledge, juxtaposing the systems-theoretical approach against Jürgen Habermas’ work on the legitimacy crisis. In concert with Luhmann, Eyal uses “validation” in the sense of defensibility to “reassure people that if they would bother to check, they would find that the particular decision was rationally taken and justified, so no need to bother!” (Eyal, 2019 , p. 88). For Luhmann, the insistence on validation is ultimately a “functionally necessary deception” that saves time and prevents discord, here given that every fact may also be interpreted differently.

Strikingly, the validity issue has become key at a time when information is openly and abundantly available and when expertise has become democratized, enabling users and other lay persons to participate in the production of evidence and render each and every piece of evidence contestable. In fact, the two prominent trends in “modern governance practices” are, according to Krick et al. ( 2019 , p. 927), a trend toward scientization and a participatory turn. The first trend is discernible in the composition of government-appointed advisory committees (a growing number of researchers), citation patterns (reference to studies, technical reports, and academic publications), and epistemic language (reference to “evidence,” “knowledge,” “data,” and “research”). The second trend implies greater public engagement in agenda setting as a result of open access to information, calling into question the corporatist or representative model of democracy (see Rommetvedt, 2017 ; Rommetvedt et al., 2012 ). Other scholars (Stehr & Grundmann, 2011 ) have labeled the second trend as a pluralization of expertise. These two trends—scientization and pluralization of expertise—have triggered a third trend that has only recently been discussed (see Lubienski, 2019 ): a surplus of evidence. Taken together, all three trends account for the fact that references, that is, the sources of information used to validate the evidence, have gained authoritative status.

Arguably, referencing other texts as an instrument of validation of one’s assertions has become an object of intense scrutiny, including in bibliometric network analyses. The pressure to disclose the sources of information that were used to produce evidence is discernible in the ever-increasing number of references listed in Green Papers. The Green Papers of the Norwegian Official Commissions (NOUs; Norges offentlige utredninger ) and the White Papers of the Ministry of Education and Research of Norway make for good cases for demonstrating the trend over time. The relevant papers of the 1996 School Reform in Norway made only sparse use of references, many of which were either embedded in the text or listed as footnotes. Twenty years later, however, there were 246 references on average per relevant Green or White Paper for the 2020 Curriculum Renewal Reform (Baek et al., 2018 ).

Paradoxically, the proliferation of evidence-based policy planning has added fuel to the crisis of expertise (see Eyal, 2019 ). Not only has science become politicized and politics scientized, but science has also become demystified in front of everyone’s eyes:

[T]he very discourse on expertise increases uncertainty and threatens legitimacy because now the public is witness to controversies between scientists. (Eyal, 2019 , p. 102)

In effect, the proliferation of evidence-based policy planning has brought to light that evidence is considered not more, and not less, than a subjective assertion backed by information. The boom has generated a surplus of evidence to the extent that there is now the challenge of how to weed out evidence based on relevance and credibility criteria. Concretely, in the wake of complexity reduction, we are witnessing a hierarchization of information (very often with randomized controlled trials on the top and qualitative data on the bottom), rendering some types of evidence more relevant than others. At the same time, the disclosure of the source of information to make a case for the credibility of the evidence, that is, the reference, has become as important, if not more so, than the information itself. In fact, the legitimacy of the assertion rests in great part on the source of the information itself. For example, a reference here and there to OECD studies has become a sine qua non for policy analysts in Europe because the OECD is seen, in the Foucauldian sense, as the founder of discursiveness for a very special kind of policy knowledge that ranks at the top in the hierarchy of evidence, one that operates with numbers and draws on international comparisons to enforce a political program of accountability. Ydesen ( 2019 ) has convincingly documented the rise of the OECD as a global education governing complex that uses a range of policy instruments (PISA, Education at a Glance, country reports, etc.) to diagnose and monitor national developments and advance the global solutions of a particular kind for national reforms.

There are many reasons why OECD studies are attractive for government officials (see Martens & Jakobi, 2010 ; Niemann & Martens, 2018 ). Espeland ( 2015 ) and Gorur ( 2015 ) masterfully observe the advantages of numbers over complex narratives because one may attach one’s own narratives to numbers. What is especially appealing to policy actors are OECD-type studies, that is, statistics, scores, ranking, and benchmarks based on international comparisons or on comparisons over time. Novoa and Yariv-Mashal dissect the politics of international comparison and examine how:

[T]his ongoing collection, production and publication of surveys leads to an ‘instant democracy’, a regime of urgency that provokes a permanent need for self-justification. ( 2003 , p. 427)

Espeland ( 2015 , p. 56) explains the dual process of simplification and elaboration involved in using numbers. In the first step, numbers “erase narratives” by systematically removing the persons, institutions, or systems being evaluated by the indicator and the researcher doing the evaluation. This technology of simplification stimulates narratives, or as Espeland astutely observes:

If the main job of indicators is to classify, reduce, simplify, and make visible certain kinds of knowledge, indicators are also generative in ways we sometimes ignore: the evoke narratives, stories about what the indicators mean, what their virtues or limitations are, who should use them to what effect, their promises, and their failings. ( 2015 , p. 65)

Scholars in comparative policy studies have started to explore why PISA and other international large-scale student assessments are so attractive to policy actors and politicians (Addey et al., 2017 ; Pizmony-Levy, 2018 ). A few studies focused on the “narrative evoking” phase (Espeland, 2015 , p. 65) of such studies have dissected what national governments interpret or project onto OECD reports or other international comparative studies based on their own policy context and agenda (Waldow & Steiner-Khamsi, 2019 ).

Reference Societies in Comparative Education Research

In addition to the governance-by-numbers argument presented above, for many countries, the OECD represents an attractive geo-political space inhabited by people in 36 high-income economies. Therefore, a recourse to OECD publications may be seen both as an affirmation of the affiliation and an acknowledgment of the OECD as a “reference society” (Bendix, 1978 , p. 292) or rather “[transnational] reference space” toward which national governments orient themselves or aspire to belong. In fact, in all five Nordic countries of the POLNET study, OECD publications represent the most cited international texts (see Chap. 11 ). This may come as a surprise for a non-Nordic audience because one would expect competition between two dominant policy discourses in the five countries of the POLNET study: the Nordic reference space, which traditionally has had a strong commitment to equity, and the OECD reference space, which has a mission to advance economic growth.

Thus, in comparative policy studies, the term “reference” also carries a spatial, geo-political, or epistemological connotation. It is used in connection with “reference society” or “reference space.” The reference as a validation instrument and the reference as a point of epistemological orientation both share a common feature: they position the author (in the case of references) or the state (in the case of reference society) in its larger discursive space.

By now, there is a well-established tradition in comparative policy studies to draw on references as an analytical tool to situate the positionality of an actor (author, institution, government) in a broader transnational, geo-political space. This body of scholarship is closely associated with studies on the “reference society” presented by sociologist Reinhard Bendix ( 1978 , p. 292). Bendix uses the term to denote how governments used economic competitors and military rivals as reference societies for their own development. One of the examples discussed by Bendix is the fascination of Meiji-era Japan with the West.

In comparative education, the term was—according to Waldow ( 2019 )—first introduced by Butts ( 1973 ), associate dean and professor of Teachers College, Columbia University. Butts observes that the governments of developing countries frequently used a specific educational system in the Global North as a model for emulation. That country’s path to “modernization” served government officials in the Global South as a reference for educational reforms in their country. It is important to bear in mind here that during Butts’ time, transnational networks and dependencies established during colonial times had endured into the present and determined in great part the choice of reference societies. Another noted historian and comparativist, David Phillips, first coined the term “cross-national policy attraction” to denote the keen interest of nineteenth-century British government officials in the educational reforms of Germany (see Ochs & Phillips, 2002 ). Both Butts and Phillips use records of study visits and government reports as sources for their analyses of cross-national attraction or policy borrowing.

Similar in the conceptual framework but different in terms of unit analysis, Schriewer and Martinez ( 2004 ) use bibliometric data to examine the use of reference societies in “educational knowledge,” as reflected in the publications of educational research journals; they wonder whether educational researchers in their sample of three countries draw on similar or different bodies of knowledge or texts. They purposefully use a time period of 70 years to see whether a convergence toward a single international canon of scientific educational knowledge, here interpreted as internationalization or globalization, has occurred. Concretely, they examine the references listed in flagship educational research journals in three countries (Spain, Russia/Soviet Union, PR China) and code them in terms of the national origin of the referenced authors. Rather than detecting a pattern of steady internationalization toward a single body of internationally acclaimed authors, they notice considerable fluctuation regarding the space allocated to international scholarship, as measured in the number and type of foreign bibliographical references made in the journal articles of the three countries. They find that the “socio-logic” (particularly political developments in a given country) was a better predictor of receptiveness toward international scholarship than an external logic as manifested in the ever-expanding transnational network of educational researchers.

In fact, the era of the greatest convergence regarding educational knowledge was in the 1920s and 1930s, when educational researchers in Spain, the Soviet Union, and China were drawn to the work of John Dewey. Once that brief period was over, Dewey was dropped from the reference list in Soviet educational journals and replaced by Nadezhda Krupskaya (Lenin’s wife). It is striking that against all expectations of globalization or international convergence theorists, educational knowledge in these three countries did not become more internationalized until after the mid-1980s, when all three opened their ideological boundaries and increased international cooperation. Even though Schriewer and Martinez’s ( 2004 ) justification for their case selection leans on a problematic notion of culture and “civilization,” the design and methodology of the study is compelling and well-suited for analyzing international convergence/divergence processes in educational research.

The link between the reference society and political change has also been well documented in comparative education research. Examples include two cases of a radical change in reference societies as a result of fundamental political changes in post-Soviet Latvia and post-socialist Mongolia, respectively. Silova ( 2006 ) examines the erasure of Soviet references and their subsequent replacement with Western European references. She interprets the shift from the Soviet to the Western European reference system as a marker for the new geo-political educational space that Latvia politically and economically had been aspiring to inhabit at the turn of the millennium. What is fascinating about this particular change of political allies is that it has merely affected the discursive level, not the practice of separate schooling. The separation of school systems, one for Latvian speakers and another for Russian and other ethnic speakers, continues to exist; however, segregated schools are no longer seen as “sites of occupation” but are now being reframed as “symbols of multiculturalism.” The list of comparative policy studies on reference societies is too long to present in an exhaustive manner. In our own study on Mongolia (Steiner-Khamsi & Stolpe, 2006 ), we observe the discursive ruptures and reorientation in terms of reference spaces that accompanied the political changes, notably the replacement of the Communist Council for Mutual Economic Assistance with the Asian Development, World Bank, and other post-communist international aid agencies.

Strikingly, studies on reference societies and cross-national policy attraction have experienced a revitalization of a special sort in recent years with the fast advance of international large-scale student assessments (ILSAs) used in many countries as a policy tool for governance by numbers (see Carvalho & Costa, 2015 ; Volante, 2018 ; Waldow & Steiner-Khamsi, 2019). Preoccupation with what league leaders (Finland, Shanghai, Singapore, etc.) have “done right” has generated new momentum for policy borrowing research. Precisely at a stage in policy borrowing research when scholars have put the study of cross-national policy attraction to rest and instead directed their attention at the ubiquitous diffusion processes of global education policies in the form of “best practices” or “international standards” vaguely defined, the cross-national dimension—and by implication the focus on the nation-state and its national policy actors—has regained importance in ILSA policy research.

In the case of PISA (Programme of International Student Assessment), the preoccupation of national policy actors is, at least rhetorically, on how their own system scores compared with others and what there is to “learn” from the league winners, league-slippers, and league-losers, in terms of PISA’s twenty-first-century skills. Because policy actors often attribute “best practices” to particular national educational systems, the national level regained importance as a unit of analysis. Therefore, ILSA policy researchers found themselves in a position of having to bring back the focus on national systems, a unit of analysis criticized as “methodological nationalism,” which, if used naively, is a cause for concern because of its homogenizing effects (see Giddens, 1995 ; Wimmer & Glick Schiller, 2003 ; Robertson & Dale, 2008 ).

Research on reference societies has also been refined over the past few years in other ways. For example, intrigued by negative media accounts in Germany about the PISA league leader Shanghai (during the 2012 PISA round), Waldow scrutinizes the policy usage of “negative reference societies” ( 2016 ) or “counter-reference societies,” respectively. The concept of a reference or counter-reference society is based on commensurability. How do national policy actors make the educational systems of league winners appear to be comparable to their own educational system in a way that can suggest that lessons could be drawn? Vice versa, how do they manage to make two educational systems incommensurable and incomparable to avoid lesson-drawing? The disbelief or the downplaying, respectively, of Chinese success in ILSAs, notably in the PISA rounds of 2012 and 2018, is comparable to earlier stereotypical accounts of Japanese or pan-Asian education.

Similar to the US media accounts of A Nation at Risk (1983) in which American policy analyses attempted but ultimately failed to persuade Americans of the great benefits of the German and Japanese educational systems, the education systems of Beijing, Shanghai, Jiangsu, and Zhejian are, despite “PISA success,” hardly used as models for emulation in Western countries. As with the A Nation at Risk report, the common reaction to Chinese success reflects a “yes, but …” attitude (see Cummings, 1989 , p. 296): even though there is a general agreement about the outstanding student performance in ILSAs in Hong Kong, Japan, Korea, Macao, Singapore, and select cities of PR China, there are too many negative stereotypes associated with education in these locations to assign them reference or emulation status. In fact, the exaggerated statements or myths about “Asian education” include images of overly ambitious mothers (“tiger mothers”), excessive use of cram schools, competition and suicide among students, elitist higher education, and social inequality. More often than not, the educational systems in Asia are politically instrumentalized as a counter-reference, that is, examples of how educational systems should not be developed.

Let us now circle back to the five-country study at hand. In the Nordic region, there is an elephant in the room in the broader ILSA space, making one wonder the following: Do the other countries of the region (Denmark, Iceland, Norway, and Sweden) consider the educational system of Finland (a PISA league leader) as a reference or a counter-reference for educational reform in their own system, or are they indifferent toward lesson-drawing from Finland? The coding of the references by their country of publication and the qualitative analysis of thematic cross references make it possible to examine the fascinating question of a reference society within the Nordic education space (see Sivesind, 2019 ; Chap. 12 ).

Clearly, the bibliometric analyses presented in this volume demonstrate that OECD publications eclipse studies from Finland. Two possible yet inconclusive interpretations lend themselves to further investigation: either “Finnish success” is acknowledged but rendered irrelevant for one’s own national context (the “yes, but …” attitude explained earlier), or Finnish success is, for a variety of reasons, including linguistic ones, referenced via an authoritative source of information: OECD publications.

Expertise-Seeking Arrangements in Policy Making

In his dissertation research, Baek ( 2020 ) coins the term “expertise-seeking arrangements,” which captures very well the dilemma of governments in an era of evidence-based policy making: Where and how do they seek advice for policy analysis, evaluation, and formulation? Given that “the authority of experts is destabilized” (Eyal, 2019 , p. 102) in an era in which scientific evidence production is easily demystified and an ever-increasing number of individuals, including concerned citizens and other laypersons, lay claim to expertise, the question of governments’ expertise-seeking arrangements is taking center stage.

Eyal presents a typology of responses to the legitimation crisis, which is reproduced in Table 2.1 below. His focus is on “regulatory science” or the “interface between scientific research, law and policy” (see Eyal, 2019 , p. 7f.).

The first strategy of the state is to pretend that science is purified from politics by pursuing “mechanical objectivity ” (Eyal, 2019 , p. 115), as reflected in references to scores, rankings, numbers, impact evaluations, and quantifiable comparisons. In our case, references to OECD studies, evaluations, and ILSAs belong to this category.

The second strategy of inclusion is to acknowledge that science is politicized—or as Latour has eloquently put it, “Science is not politics. It is politics by other means” ( 1984 , p. 229)—and, therefore, includes laypersons, interest groups, and other engaged citizens into the advisory bodies of the government.

The third strategy of exclusion is to decouple science from politics and generate “gate-keeping mechanisms designed to maintain an artificial scarcity of expertise” (Eyal, 2019 , p. 105; see also Weingart, 2003 ); these are typically academic associations (academy of sciences) or professional associations that claim exclusive expertise and advise the government.

The fourth and final strategy of outsourcing represents another functional differentiation process. It decouples science from politics to the extent that it delegates research and policy formulation to outside groups, think-tanks, or semiprivate entities. Eyal contends that the fourth strategy is oftentimes a reaction to failed attempts of the state to generate trust or being inclusive, which would be pursued in the first three strategies. In particular, the fourth strategy is often a response to the critique that the second strategy—the appointment of ad hoc advisory commissions or government-appointed expert commissions—are merely meant for window-dressing and rarely impact the ultimate policy decisions and formulations prepared by government officials.

Another useful typology of “advisory system activity” has been developed by Craft and Howlett ( 2013 , p. 193ff.). In general, they find a trend toward the inclusion of nonstate actors (think-tanks, open data citizen engagement driven policy initiatives/web 2.0, etc.) and international actors (e.g., OECD, ILO, UN organizations) as policy advisors. This is in stark contrast to traditional advisory systems, which mainly have drawn on national advisory bodies, including statistical offices, strategic policy units within the government, or government-appointed ad hoc commissions.

Naturally, the changing nature of the relationship between politics and science has preoccupied comparative policy studies for a while. One of the early, more important comparative studies exploring the interpenetration of the two function systems is the research project “The role of knowledge in the construction and regulation of health and education policy in Europe: convergences and specificities among nations and sectors,” abbreviated as Know&Pol and funded in the Sixth Framework Programme of the European Commission (see, e.g., Fenwick et al., 2014 ). Knowledge-based regulation, which is analyzed in the Know&Pol research project, has fundamentally changed the role of the state from one that runs schools to one that establishes learning standards and monitors learning outcomes, thereby enabling a multitude of providers, including businesses, to enter the school market. Over the past 20 years or so, the private sector has become not only a major provider of education, but also a key policy actor, lobbying for reforms that further restrict the role of the state in the education sector (Verger et al., 2017 ).

As mentioned above, knowledge-based regulation has also enlarged the radius of individuals contributing to policy-relevant educational knowledge. An early indication of changes in knowledge production and sharing is the open-access policies that both governments and research councils have put in place recently. System theorists Peter Weingart and Justus Lentsch ( 2008 ) consider such open-access policies to be part and parcel of a democratization of expertise; here, the relationship between science and politics has experienced three distinct shifts over the past 70 years ( 2008 , p. 207 ff.). During the early period of scientific policy advice (1950s–1970s), the ad hoc expert commissions insisted on being autonomous and independent from governments. As a corollary, their reports amassed foundational scientific knowledge that policy actors could or could not use, respectively. In a second phase, the commissions became increasingly politicized (1970s to 1990s) because they were charged with the task of producing policy-relevant scientific knowledge. In the current third phase, the governments in many countries have experienced a shift from “knowledge-based legitimacy” to “participation-based legitimacy.” This also applies to government-appointed ad hoc expert commissions. Governments are under pressure to “democratize” scientific policy advice by (i) providing open access to reviews and expertise, (ii) expanding the definition of “experts” (including nowadays, both producers and consumers), and (iii) insisting that the knowledge products are useful, that is, provide a clear foundation for stop/go policy decisions.

In the five participating countries of the Nordic region, there is a wide array of expertise-seeking arrangements that these countries’ governments have put in place (see Chap. 10 ). The Eurydice Report ( 2017 ), Support Mechanisms for Evidence-Based Policy-Making in Education , is incomplete (data on Iceland is missing) and too imprecise to provide any useful clues for a categorization of expertise-seeking arrangements. For example, the government-appointed expert commissions in Norway (NOUs) and Sweden (SOUs) that amass evidence to substantiate their evaluation of past reforms and their recommendations for new directions are not mentioned. As documented in the OECD study on policy advisory systems (OECD, 2017 ), in all five countries, there is a commitment to evidence-based policy planning (which in some countries is inscribed in law), an extensive stakeholder review, or a “hearing” process in which draft versions of new policy are opened up for public consultation.

The type of expertise-seeking arrangement in each of the five countries needs to be kept in mind when interpreting the role of experts in producing evidence for policy making. It may be useful to draw on an existing typology of such arrangements. For example, Weingart and Lentsch identify six types of commissions that provide scientific advice for policy making ( 2008 , Chap. 2 ): (i) policy-domain-specific advisory councils, (ii) expert commissions for risk management, (iii) policy-specific expert commissions, (iv) ad hoc commissions, (v) enquete commissions, and (vi) sector research.

The typology may be used to categorize the five types of expertise-seeking arrangements in the countries of the Nordic region. According to the typology of Weingart and Lentsch ( 2008 ), the government-appointed “official commissions” in Norway and Sweden (NOUs and SOUs, respectively), which prepare and help legitimize policy decisions, fall into the categorization of “ad hoc commissions.” In Denmark, the School Council, which was established in 2006, serves to advise the ministry on topics related to elementary school (see Chap. 4 ). In Finland, the expertise-seeking arrangement is multisited or hybridized, according to Holli and Turkka ( 2021 ). The government-appointed ad hoc commissions, which exist in Norway and Sweden, were abolished in 2003 and replaced with broad-based working groups. The representation of academics declined over time and constituted only 4.7% of all working group members in 2015. At the same time, the Government of Finland pluralized the policy advisory system:

[T]he policy advisory system of Finland shows signs of hybridisation, as the channels and organization of policy advice have pluralised and advice has taken new forms. (Holli & Turkka, 2021 , p. 58 [translation by the authors])

The partial externalization of policy advice is manifested in the rise of “state investigators” or consultants who are hired to produce government-commissioned reports to a general outsourcing of policy research. According to the authors, Finland has created a “research market,” where the state buys the research it needs for policy preparation. In Iceland, finally, the composition of advisory bodies is strictly regulated in terms of gender and political parties to ensure an inclusive consultative process. According to the OECD survey on policy advisory systems, which is carried out in 17 countries, including the 5 Nordic countries studied here, the policy advisory system in Iceland requires that at least 40 of the members of ad hoc advisory commissions are female and that all political parties are represented (OECD, 2017 ).

The OECD has formulated five quality standards for policy advisory systems: adaptability, transparency, autonomy, inclusiveness, and effectiveness. The OECD 17-country study (OECD, 2017 ) presents a positive assessment of the ad hoc advisory committees found in the Nordic region:

Ad hoc advisory bodies […] are often used by governments to gather evidence-based answers to particular questions relatively quickly. They often serve as a “fast track” and specialized option for governments to obtain advice. The Nordic countries have well-established traditions of creating ad hoc bodies to enhance the adaptability of the system. (OECD, 2017 , p. 17)

Implications for the Five-Country Bibliometric Network Analyses

In the parallel universe of “gray literature” or technical reports, which are often commissioned by international organizations, there is an interesting discussion unfolding on the rapid spread of “global public goods” (GPGs) or global knowledge banks. GPGs include, for example, openly accessible international toolkits, documents, studies and databanks, training modules, good practices, and global monitoring reports. Clearly, the GPGs are openly accessible information, nowadays often backed up with numbers that any local, national, or international organization may use to back up their production of evidence. The rapid spread of GPGs was addressed by a few scholars around the turn of the millennium (notably Stone, 2000 ), but curiously, the discussion has not yet gained traction in academic debates. Hence, a brief summary of the debates, carried out in the context of development studies, may be in order here.

Within development studies, the discussion is now bifurcating in at least two different directions. One group of authors makes the argument for more funding for GPGs produced by a more diverse body of researchers (based in the Global North and the Global South) and another group critically examines the uptake of GPGs for national policy making at the national level (see Vasquez Cuevas, 2020 ).

In some countries, a wide range of propositions have been made about how to remedy the shortfalls related to global agenda setting, channeling of aid, and GPGs (see Schäferhoff & Burnett, 2016 ). Some suggestions entail more funding at the global level, whereas others notice that the production of GPGs is mostly done by a few global actors (OECD, the World Bank, the UN system) at the expense of national research institutions outside of North America, Europe, and Australia. This applies in particular to think-tanks, research institutions, and universities in the Global South, whose knowledge products are rarely taken up at the global level. Examples of more funding to the Global South include Oxfam’s early suggestion to eliminate one-size-fits-all benchmarking processes and dedicate three grants for capacity building to recipient governments and civil society organizations of low- and lower-middle-income countries (Oxfam, 2010 ).

For a while, the question arose whether the World Bank, UNESCO institutions, UNICEF, Global Partnership for Education, or other international organizations should earmark funds for research capacity building and policy analysis. One of the early suggestions was to increase funding for the global and regional agencies of UNESCO and UNICEF to advance cross-country sharing of knowledge on education and development. In addition to statistics, the UN organizations would use the funds to disseminate knowledge derived from research and from global sharing of experience. Others found the World Bank to be ideally suited for helping expand research funding and activities given its commitment to evidence-based policy making; they recommended that researchers at the World Bank would work more closely with other staff for country-level policy advice (Clemens & Kremer, 2016 ). Unsurprisingly, the lively debate in development studies is on whose knowledge is made publicly available at the global level and whose knowledge is confined to the national boundaries of its producers.

In OECD countries, the debate over the asymmetry of global knowledge production and update seems to center more on the language of publication (English vs. all other languages) rather than on the center/periphery differentiation in an unequal world system. Some countries in the Nordic region (Sweden and Norway in particular) require that all public documents be made openly accessible. The transparency standard of policy advisory systems, which are forcefully promoted by the OECD ( 2015 , 2017 ), is also practiced in the other three Nordic countries.

The open-access policies and practices have both enabled and exacerbated the “participatory turn” or the “pluralization of expertise” (see Krick et al., 2019 ), which has been explained above. The decline in corporatism or interest group representation in policy advisory systems is but one of the manifestations of this trend. Another manifestation is the surplus of evidence.

In the US context, Lubienski ( 2019 ) contends that there is not a scarcity but rather a “surplus of evidence.” In such a “marketplace of ideas,” there is ample opportunity for new, nonstate actors—specifically the private sector—to serve as intermediaries between research production and policy making:

Into the chasm between research production and policy-making, we are seeing the entrance of new actors—networks of intermediaries—that seek to collect, interpret, package, and promote evidence for policymakers to use in forming their decisions. (Lubienski, 2019 , p. 70)

Indeed, two decades after neoliberal calls for less politics and more scientific rationality in the policy process, we are now entering a new phase in policy making: the stage of surplus of evidence in which calls for “actionable research,” “policy-relevant research,” or “what works” studies are being heard. At the 2015 Public Governance Ministerial Meeting in Helsinki, the ministers from OECD countries agreed (OECD, 2015 , p. 3) that evidence alone is not sufficient. Evidence needs to be policy relevant, robust, and comparable:

We acknowledge the importance of evidence as a critical underpinning of public policies and recognize the need for a continuous effort to develop policy-relevant evidence on government performance that is robust and comparable. “What Works” initiatives are an example of how to ensure systematic assessment and leverage the stock of information on good practices available at the international level on policy impact. (OECD, 2015 , p. 3)

In the Nordic policy context, the OECD plays a major role as a transnational policy advisor and standard setter. During the first phase of evidence-based policy planning, the OECD advanced the notion of autonomous policy advisor systems that produce, independent of the state, evidence. In today’s stage of evidence overproduction, the OECD offers itself as an interpreter of evidence by selecting from the marketplace of ideas those that are actionable and in line with the broader political program of accountability.

The proliferation of GPGs and the overproduction of evidence make it pressing to investigate the changing role of government-appointed advisory commissions in the production and interpretation of evidence. A host of research questions open up once we acknowledge that the presentation of “facts” (information) and transformation of these facts into evidence rests on a subjective selection process that reflects the frame of reference or broader discursive orientation of the author.

Based on the elaborations presented above, we have several research questions that lend themselves to a systems-theoretical preoccupation with evidence-based policy planning: First, what sources of information do government-appointed advisory panels consider credible and, therefore, select to reduce uncertainty and generate trust? Second, what kind of hierarchization of evidence do government-appointed advisory panels generate to reduce complexity? Third, bearing in mind the three most common types of externalization, what type of externalization do the cited texts represent: reference to (i) tradition or values (e.g., Nordic value of equity), (ii) organization (e.g., reference to laws and regulations), or (iii) scientific rationality (e.g., studies and evaluations). Fourth, in which broader epistemic community, or rather political reference space, do the authors situate themselves? Finally, given the academization of government-appointed advisory commissions in some countries (Christensen & Hesstvedt, 2018 ), as a result of which the number of academics serving as members increased at the expense of interest group representatives, the phenomenon of structural coupling between science and politics (Steiner-Khamsi et al., 2019 ; Weingart, 2003 ; Stehr & Grundmann, 2011 ; see also Chap. 10 ) offers itself as an object of empirical scrutiny.

This chapter has attempted to demonstrate how a bibliometric network analysis may be used as a method of inquiry for understanding legitimization processes in evidence-based policy making. Drawing our attention to how uncertainty is reduced and trust in evidence is created is essential in an era in which there is a surplus and, by implication, competing notions of evidence. A bibliometric investigation of the references in a text provides important clues about the (i) selection of sources of information, (ii) hierarchization of evidence, (iii) and type of externalization made by an author to leverage authority for the claims made in the text. A network analysis of the references further helps complicate the findings. In fact, the focus on relations brings to light that different authors use the same references for different purposes, that is, one and the same reference may show up in, and bridge, two different knowledge networks. Perhaps needless to reiterate, the long list of fascinating research questions, which are presented in a nonexhaustive manner in this chapter, gains additional attraction when investigated across the five different national contexts within the Nordic region.

Addey, C., Sellar, S., Steiner-Khamsi, G., Lingard, B., & Verger, A. (2017). Forum discussion: The rise of international large-scale assessments and rationales for participation. Compare, 47 (3), 434–452.

Article   Google Scholar  

Baek, C., Hörmann, B., Karseth, B., Pizmony-Levy, O., Sivesind, K., & Steiner-Khamsi, G. (2018). Policy learning in Norwegian school reform: A social network analysis of the 2020 incremental reform. Nordic Journal of Studies in Educational Policy, 4 (1), 24–37. https://doi.org/10.1080/20020317.2017.1412747

Baek, D. (2020). Knowledge utilization in education policymaking in the United States, South Korea, and Norway: A bibliometric network analysis . Doctoral dissertation, Columbia University, Graduate School of Arts and Sciences.

Google Scholar  

Bendix, R. (1978). Kings or people: Power and the mandate to rule . University of California Press.

Butts, R. F. (1973). New futures for comparative education. Comparative Education Review, 17 (3), 289–294. https://doi.org/10.2307/1186966

Cairney, P. (2015). The politics of evidence-based policymaking . Palgrave Pivot.

Carvalho, L. M., & Costa, E. (2015). Seeing education with one’s own eyes and through PISA lenses: Considerations of the reception of PISA in European countries. Discourse: Studies in the Cultural Politics of Education, 36 (4), 1–9.

Christensen, J., & Hesstvedt, S. (2018). Expertisation or greater representation? Evidence from Norwegian Advisory Commissions. European Politics and Society, 20 (1), 83–100.

Clemens, M., & Kremer, M. (2016). The new role of the World Bank [Working Paper 421]. The World Bank.

Craft, J., & Howlett, M. (2013). The dual dynamics of policy advisory systems: The impact of externalization and politicization on policy advice. Policy and Society, 32 (3), 187–197.

Cummings, W. K. (1989). The American perception of Japanese education. Comparative Education, 25 (3), 293–302.

Espeland, W. (2015). Narrating numbers. In R. Rottenburg, S. E. Merry, S.-J. Park, & J. Mugler (Eds.), The world of indicators. The making of governmental knowledge through quantification (pp. 56–75). Cambridge University Press.

Chapter   Google Scholar  

European Commission/EACEA/Eurydice. (2017). Support mechanisms for evidence-based policy-making in education [Eurydice Report]. Publications Office of the European Union.

Eyal, G. (2019). The crisis of expertise . Polity Press.

Fenwick, T., Mangez, E., & Ozga, J. (2014). World Yearbook of Education 2014. Governing knowledge: Comparison, knowledge-based technologies and expertise in the regulation of education . Routledge.

Book   Google Scholar  

Giddens, A. (1995). National state and violence . University of California Press.

Gorur, R. (2015). Producing calculable worlds: Education at a glance. Discourse: Studies in the Cultural Politics of Education, 36 (4), 578–595.

Holli, A. M., & Turkka, S. (2021). Tieteen muuttuva rooli korporatistisessa neuvonannossa. Pitkittäisanalyysi tutkijoiden asemasta ministeriöiden valmistelutyöryhmissä 1980–2018. Politiikka, 63 (1), 54–81.

Krick, E., Christensen, J., & Holst, C. (2019). Between ‘scientization’ and a ‘participatory turn’. Tracing shifts in the governance of policy advice. Science and Public Policy, 46 (6), 927–939.

Latour, B. (1984). The pasteurization of France . Harvard University Press.

Lubienski, C. (2019). Advocacy networks and market models for education. In M. P. do Amaral, M. Steiner-Khamsi, & C. Thompson (Eds.), Researching the global education industry (pp. 69–86). Palgrave.

Martens, K., & Jakobi, A. (Eds.). (2010). Mechanisms of OECD governance—International incentives for national policy making . Oxford University Press.

Niemann, D., & Martens, K. (2018). Soft governance by hard fact? The OECD as a knowledge broker in education policy. Global Social Policy, 18 (3), 267–283. https://doi.org/10.1177/1468018118794076

Novoa, A., & Yariv-Mashal, T. (2003). Comparative research in education: A mode of governance or a historical journey? Comparative Education, 39 (4), 423–438.

Ochs, K., & Phillips, D. (2002). Toward a structural typology of cross-national attraction in education . Educe.

OECD. (2015). Public governance for inclusive growth. Towards a new vision for the public sector. Public Governance Ministerial Meeting, 28 October 2015, Helsinki. OECD.

OECD. (2017). Policy advisory systems. Supporting good governance and sound public decision making .

Oxfam. (2010). Rescuing education for all. How reform of the Fast-Track Initiative should lead to a Global Fund for Education . Oxfam International.

Ozga, J. (2019). The politics of accountability. Journal of Educational Change . https://doi.org/10.1007/s10833-019-09354-2

Pizmony-Levy, O. (2018). Compare globally, interpret locally: International assessments and news media in Israel. Globalisation, Societies and Education, 16 (5), 577–596. https://doi.org/10.1080/14767724.2018.1531236

Robertson, S. L., & Dale, R. (2008). Researching education in a globalizing era: Beyond methodological nationalism, methodological statism, methodological educationism and spatial fetishism. In J. Resnik (Ed.), The production of educational knowledge in the global era (pp. 19–32). Sense Publishers.

Rommetvedt, H. (2017). Scandinavian corporatism in decline. In K. Oddbjoern (Ed.), The Nordic models in political science (pp. 171–192). Fagbokforlaget.

Rommetvedt, H., Thesen, G., Christiansen, P. M., & Norgaard, A. S. (2012). Coping with corporatism in decline and the revival of parliament: Interest group lobbyism in Denmark and Norway, 1980–2005. Comparative Political Studies, 46 (4), 457–485.

Schäferhoff, M., & Burnett, B. (2016). Rethinking the financing and architecture of global education. Background Paper for the Education Commission . Education Commission.

Schriewer, J., & Martinez, C. (2004). Constructions of internationality in education. In G. Steiner-Khamsi (Ed.), The global politics of educational borrowing and lending (pp. 29–53). Teachers College Press.

Silova, I. (2006). From sites of occupation to symbols of multiculturalism. Reconceptualizing minority education in post-Soviet Latvia . Information Age Publisher.

Sivesind, K. (2019). Nordic reference societies in school reforms in Norway: An examination of Finland and the use of international large-scale assessments. In F. Waldow & G. Steiner-Khamsi (Eds.), Understanding PISA’s attractiveness: Critical analyses in comparative policy studies (pp. 89–107). Bloomsbury.

Stehr, N., & Grundmann, R. (2011). Experts: The knowledge and power of expertise . Routledge.

Steiner-Khamsi, G., Karseth, B., & Baek, C. (2019). From science to politics: Commissioned reports and their political translation into White Papers. Journal of Education Policy, 35 (1), 119–144.

Steiner-Khamsi, G., & Stolpe, I. (2006). Educational import in Mongolia: Local encounters with global forces . Palgrave Macmillan.

Stone, D. (2000). Banking on knowledge: The genesis of the Global Development Network . Routledge.

Vasquez Cuevas, M. (2020). Global monitoring of national development: Coercive or collaborate? NORRAG Special Issue 03.

Verger, A., Steiner-Khamsi, G., & Lubienski, C. (2017). The emerging global education industry: Analyzing market-making in education through market sociology. Globalisation, Societies and Education, 15 (3), 325–340.

Volante, L. (2018). The PISA effect on global educational governance . Routledge.

Waldow, F. (2016). Das Ausland als Gegenargument: Fünf Thesen zur Bedeutung nationaler Stereotype und negativer Referenzgesellschaften [Foreign countries as a counterargument: Five propositions regarding the significance of national stereotypes and negative reference societies]. Zeitschrift für Pädagogik, 62 (3), 403–421.

Waldow, F. (2019). Introduction: Projection in education policy-making. In F. Waldow & G. Steiner-Khamsi (Eds.), Understanding PISA’s attractiveness: Critical analyses in comparative policy studies . Bloomsbury.

Waldow, F., & Steiner-Khamsi, G. (Eds.). (2019). Understanding PISA’s Attractiveness: Critical Analyses in Comparative Policy Studies . London: Bloomsbury.

Weingart, P. (2003). The paradoxes of expert advising. In G. Bechmann & I. Hronszky (Eds.), Expertise and its interfaces. The tense relationship of science and politics (pp. 43–89). Edition Sigma.

Weingart, P., & Lentsch, J. (2008). Wissen, Beraten, Entscheiden. Form und Funktion wissenschaf-tlicher Politikberatung in Deutschland. [Knowledge, advice, decision making. Form and function of scientific policy advice in Germany]. Velbrück.

Wimmer, A., & Glick Schiller, N. (2003). Methodological nationalism, the social sciences, and the study of migration: An essay in historical epistemology. International Migration Review, 37 (3), 576–610. https://doi.org/10.1111%2Fj.1747-7379.2003.tb00151.x

Ydesen, C. (2019). The formation and workings of a global education governing complex. In C. Ydesen (Ed.), The OECD’s historical rise in education. The formation of a global governing complex (pp. 291–304). Palgrave.

Download references

Author information

Authors and affiliations.

Teachers College, Columbia University, New York, NY, USA

Gita Steiner-Khamsi

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Gita Steiner-Khamsi .

Editor information

Editors and affiliations.

Department of Education, University of Oslo, Oslo, Norway

Berit Karseth  & Kirsten Sivesind  & 

Rights and permissions

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

Reprints and permissions

Copyright information

© 2022 The Author(s)

About this chapter

Steiner-Khamsi, G. (2022). What Is in a Reference? Theoretically Understanding the Uses of Evidence in Education Policy. In: Karseth, B., Sivesind, K., Steiner-Khamsi, G. (eds) Evidence and Expertise in Nordic Education Policy. Palgrave Macmillan, Cham. https://doi.org/10.1007/978-3-030-91959-7_2

Download citation

DOI : https://doi.org/10.1007/978-3-030-91959-7_2

Published : 02 April 2022

Publisher Name : Palgrave Macmillan, Cham

Print ISBN : 978-3-030-91958-0

Online ISBN : 978-3-030-91959-7

eBook Packages : Education Education (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Last updated 09/07/24: Online ordering is currently unavailable due to technical issues. We apologise for any delays responding to customers while we resolve this. For further updates please visit our website: https://www.cambridge.org/news-and-insights/technical-incident

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings .

Login Alert

• > Effective Teaching and Successful Learning
• > References

Book contents

• Frontmatter
• Introduction
• 1 Main Features of Scientific Research on Education
• 2 Important Types of Scientific Research on Education
• 3 Main Features of Evidence-based Research on Education
• 4 Meta-Analyses on Education
• 5 A Synthesis of Over 800 Meta-Analyses Relating to Achievement
• 6 Scaffolding Effective Teaching and Successful Learning
• 7 Planning and Starting the Lesson
• 8 Presenting Knowledge and Skills – Assertive Questioning
• 9 Guided and Independent Practice
• 10 Cooperative and Project-based Learning
• 11 Feedback – Reciprocal and Informative
• Concluding Remarks: Standards Need More Evidence

Published online by Cambridge University Press:  05 June 2016

Access options

Save book to kindle.

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle .

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service .

• Inez De Florio , Universität Kassel, Germany
• Book: Effective Teaching and Successful Learning
• Online publication: 05 June 2016
• Chapter DOI: https://doi.org/10.1017/CBO9781316285596.015

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox .

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive .

• Research guides

Educational Research

Sample references, book references.

A reference for a basic book with one author:

Author, A.A. (1967). Title of book . Location: Publisher.

Gast, L.E. (2012). Mastering approaches to diversity in social work . Philadelphia: Jessica Kingsley Publishers.

A chapter in an edited book:

Author, A.A., & Author, B.B. (1995). Title of chapter or entry. In A. Editor, B. Editor, & C. Editor (Eds.), Title of book (pp. xxx-xxx). Location: Publisher.

  Example:

Kayser, K. & Johnson, J.K. (2008). Divorce. In Mizrahi, T. & Davis, T.E. (Eds.), The encyclopedia of social work (pp. 76-85).   New York: National Association of Social Workers.

Website with Authors Identified

Author, A. A., & Author, B. B. (Date of publication). Title of document . Retrieved from http://Web address

Website with No Authors

When there is no author for a web page, the title moves to the first position of the reference entry.

Example of a website without an author:

National Institute of Mental Health. (2010). Retrieved from http://www.nimh.nih.gov/index.shtml

Journal References

A basic journal reference:

Author, A. A., Author, B. B., & Author, C. C. (Year). Title of article. Title of Periodical, xx , x-xx.

An electronic journal with a DOI

Author, A.A., Author, B.B., & Author, C.C. (year). Title of article. Title of Periodical , xx , x-xx. doi:xx.xxxxxxxxxx  

Edwards, H.R. & Hoefer, R. (2010). Are social work advocacy groups using web 2.0 effectively? Journal of Policy Practice , 9 , 220-229. doi:10.1080/15588742.2010.489037

What is a DOI?

A DOI (digital object identifier) is an article identification number provides a permanent url for an electronic resource. You can usually find the DOI   located on the first page of the electronic journal article, near the copyright notice.

Can't find the DOI of an electronic journal? Cite like this:

Author, A.A. (year) Title of article. Title of Periodical, x , xx-xx. Retrieved from: http://www.xxxx.ccc

Newspaper Articles

Print newspaper article:

Author, A. (1983, September 30). Title of article. Newspaper Title , pp. A1, A4.

Online newspaper article:

Author, A.A. (2009, December 11). Title of article. Newspaper Title . Retrieved from http://www.nytimes.com

Wu, J.Q. (2010, August 22). Controlling emotion key to this game.  The Boston Globe. Retrieved from http://www.boston.com

• << Previous: APA Style
• Next: Formatting Basics >>
• Last Updated: Jul 11, 2024 2:45 PM
• Subjects: Education
• Tags: education , education_databases , educational_research , research_in_education , researching_education

Reference management. Clean and simple.

How to format your references using the Educational Research citation style

This is a short guide how to format citations and the bibliography in a manuscript for Educational Research. For a complete guide how to prepare your manuscript refer to the journal's instructions to authors .

• Using reference management software

Typically you don't format your citations and bibliography by hand. The easiest way is to use a reference manager:

• Journal articles

Those examples are references to articles in scholarly journals and how they are supposed to appear in your bibliography.

Not all journals organize their published articles in volumes and issues, so these fields are optional. Some electronic journals do not provide a page range, but instead list an article identifier. In a case like this it's safe to use the article identifier instead of the page range.

• Books and book chapters

Here are examples of references for authored and edited books as well as book chapters.

Sometimes references to web sites should appear directly in the text rather than in the bibliography. Refer to the Instructions to authors for Educational Research .

This example shows the general structure used for government reports, technical reports, and scientific reports. If you can't locate the report number then it might be better to cite the report as a book. For reports it is usually not individual people that are credited as authors, but a governmental department or agency like "U. S. Food and Drug Administration" or "National Cancer Institute".

• Theses and dissertations

Theses including Ph.D. dissertations, Master's theses or Bachelor theses follow the basic format outlined below.

• News paper articles

Unlike scholarly journals, news papers do not usually have a volume and issue number. Instead, the full date and page number is required for a correct reference.

• In-text citations

References should be cited in the text by name and year in parentheses :

Here are examples of in-text citations with multiple authors:

• Two authors: (Burghardt and Ajtai 2012)
• Three authors: (Zhang, Keleshian, and Sachs 2001)
• 4 or more authors: (Wang et al. 2017)
• About the journal

• Other styles
• Integration, the VLSI Journal
• Environment and Planning B, Planning & Design

SDG4 – Quality Education

ISBN : 978-1-78769-426-2 , eISBN : 978-1-78769-423-1

Publication date: 9 November 2018

Ferguson, T. , Iliško, D. , Roofe, C. and Hill, S. (2018), "References", SDG4 – Quality Education ( Concise Guides to the United Nations Sustainable Development Goals ), Emerald Publishing Limited, Leeds, pp. 101-120. https://doi.org/10.1108/978-1-78769-423-120181015

Emerald Publishing Limited

Copyright © 2019 Therese Ferguson, Dzintra Iliško, Carmel Roofe, and Susan Hill

Adger, Benjaminsen, Brown, & Svarstad (2001) Adger , W. N. , Benjaminsen , T. A. , Brown , K. , & Svarstad , H. ( 2001 ). Advancing a political ecology of global environmental discourses . Development and Change , 32 ( 4 ), 681 – 715 .

Agrawal (1995) Agrawal , A. ( 1995 ). Dismantling the divide between indigenous and scientific knowledge . Development and Change , 26 ( 3 ), 413 – 439 .

Ainscow & Cesar (2006) Ainscow , M. , & Cesar , M. ( 2006 ). Inclusive education ten years after Salamanca: Setting the agenda . European Journal of Psychology of Education , 21 , 231 – 238 .

Ballard (2005) Ballard , D. ( 2005 ). Using learning processes to promote change for sustainable development . Action Research , 3 ( 2 ), 135 – 156 .

Banks (2008) Banks , J. A. ( 2008 ). An introduction to multicultural education . Boston, MA : Pearson .

Berry (1994) Berry , T. ( 1994 ). The great work: Our way into the future . New York, NY : Bell Tower .

Bertelsmann Stiftung and Sustainable Development Solutions Network (2017) Bertelsmann Stiftung and Sustainable Development Solutions Network . ( 2017 ). SDG index and dashboards report. Retrieved from http://www.sdgindex.org/assets/files/2017/2017-SDG-Index-and-Dashboards-Report--full.pdf

Berziņa (2010) Berziņa , Z. ( 2010 ). Mentoring for promoting collaborative culture in inclusive school. Summary of the doctoral dissertation . Saule, Latvia : Daugavpils University .

Berzina (2011) Berzina , Z. ( 2011 ). School-based mentoring for professional development of inclusive school teachers . Journal of Teacher Education for Sustainability , 13 ( 1 ), 72 – 83 . doi: 10.2478/v10099-011-0006-0

Blaine (2018) Blaine , B. ( 2018 ). Fix family life, fix crime. The Gleaner , January 14. Retrieved from http://jamaica-gleaner.com/article/commentary/20180114/betty-ann-blaine-fix-family-life-fix-crime

Booth (2011) Booth , T. ( 2011 ). The name of rose: Inclusive values into action in teacher education . Prospects , 41 ( 3 ), 303 – 318 .

Booth & Ainscow (2002) Booth , T. , & Ainscow , M. ( 2002 ). Index of inclusion. Developing learning and participation at schools . Centre of Studies on Inclusive Education . Retrieved from http://www.eenet.org.uk/resources/docs/Index%20English.pdf

Bourke & Carrington (2007) Bourke , P. , & Carrington , S. ( 2007 ). Inclusive education reform: Implications for teacher aides . Australasian Journal of Special Education , 31 ( 1 ), 15 – 24 .

Bowers (2001) Bowers , C. A. ( 2001 ). Educating for eco justice and community . Athens : The University of Georgia Press .

Bradbury (2003) Bradbury , H. ( 2003 ). Sustaining inner and outer worlds: A whole- system approach to developing sustainable business practices in management . Journal of Management Education , 27 ( 2 ), 172 – 187 .

Bransford, Brown, & Cocking (2000) Bransford , J. , Brown , A. , & Cocking , R. (Eds.). ( 2000 ). How people learn: Brain, mind, experience, and school-expanded edition . Washington, DC : National Academies Press .

Brundrett & Duncan (2010) Brundrett , M. , & Duncan , D. ( 2010 ). Leading curriculum change . Retrieved from http://dera.ioe.ac.uk

Buckler & Creech (2014) Buckler , C. , & Creech , H. ( 2014 ). Shaping the future, we want: UN Decade of Education for Sustainable Development (2005–2014) final report . Paris, France: UNESCO. Retrieved from http://unesdoc.unesco.org/images/0023/002301/230171e.pdf

Collins-Figueroa, Sanguinetti Phillips, Foster-Allen, & Falloon (2008) Collins-Figueroa , M. , Sanguinetti Phillips , G. , Foster-Allen , E. , & Falloon , C. ( 2008 ). Advancing Jamaican formal education through environmental education for sustainable development . Caribbean Journal of Education , 30 ( 1 ), 160 – 176 .

Crawford, Schlager, Toyama, Riel, & Vahey (2005) Crawford , V. , Schlager , M. , Toyama , U. , Riel , M. , & Vahey , P. ( 2005 ). Characterizing adaptive expertise in science teaching. Report on laboratory study of teacher reasoning. Paper presented at the American Educational Research Association Annual Conference, Montreal, Canada.

Davis (2004) Davis , R. ( 2004 ). Task force on educational reform report . Retrieved from http://jis.gov.jm/estp/docs/Reports/JA%20Education%20Reform%20TaskForce%202004.pdf

Dewey (1976) Dewey , J. ( 1976 ). Individuality, equality, and superiority . In J. A. Boydston (Ed.), John Dewey: The middle works, 1899–1924 (Vol. 13 , pp. 295 – 300 ). Carbondale, IL : Southern Illinois University Press .

Doll (1996) Doll , R. ( 1996 ). Curriculum improvement: Decision-making and process . Neeham Heights, MA : Allyn & Bacon .

Down & Nurse (2007) Down , L. , & Nurse , H. ( 2007 ). Education for sustainable development networks, potential and challenge: A critical reflection on the formation of the Caribbean Regional Network . Journal of Education for Teaching International Research and Pedagogy , 33 ( 2 ), 177 – 190 .

Easton & Collins-Figueroa (2000) Easton , C. , & Collins-Figueroa , M. ( 2000 ). A national environmental education action plan in Jamaica . Environmental Education , 63 , 6 – 7 .

Education (2015) Education for All ( 2015 ). National review report: Jamaica .

EFA Global Monitoring Report (2005) EFA Global Monitoring Report . ( 2005 ). Retrieved from http://unesdoc.unesco.org/images/0023/002300/230020E.pdf

Elson, Fukuda Parr, & Vizard (2014) Elson , D. , Fukuda Parr , S. , & Vizard , P. ( 2014 ). Human rights and the capabilities approach: An interdisciplinary dialogue . London : Routledge .

Espinosa & Walker (2017) Espinosa , A. , & Walker , J. ( 2017 ). A complexity approach to sustainability. Theory and application . London : World Scientific Publication .

European University Association (2014) European University Association . ( 2014 ). Quality culture in European universities. A bottom-up approach . Retrieved from http://www.eua.be/eua/jsp/en/upload/Quality_Culture_2002_2003.1150459570109.pdf

Evans (2001) Evans , H. ( 2001 ). Inside Jamaican schools . Kingston : University of the West Indies Press .

Falkenberg & Babiuk (2014) Falkenberg , T. , & Babiuk , G. ( 2014 ). The status of education for sustainability in initial teacher education programmes: A Canadian case study . International Journal of Sustainability in Higher Education , 15 ( 4 ), 418 – 430 . doi: 10.1108/IJSHE-10–2012-0088

Ferguson (2008) Ferguson , T. ( 2008 ). ‘Nature’ and the ‘environment’ in Jamaica’s primary school curriculum guides . Environmental Education Research , 14 ( 5 ), 559 – 577 .

Forlin (2010) Forlin , C. ( 2010 ). Teacher education for inclusion . In R. Rose (Ed.), Confronting obstacles to inclusion. International responses to developing inclusive education (pp. 155 – 170 ). London : Routledge .

Freire (2001) Freire , P. ( 2001 ). Pedagogy of freedom, ethics, democracy and civic courage . Lanham, MD : Rowan & Littlefield .

Friedman, Razer, & Sykes (2004) Friedman , V. , Razer , M. , & Sykes , I. ( 2004 ). Towards a theory of inclusive practice: An action science approach . Action Research , 2 ( 2 ), 167 – 189 .

Fullan (2002) Fullan , M. ( 2002 ). Principals as leaders in a culture of change. Educational Leadership , Special Issue, May, 1–15. Retrieved from http://michaelfullan.ca/wp-content/uploads/2016/06/13396053050.pdf

Fullan (2007) Fullan , M. ( 2007 ). Change forces: Probing the depths of educational reform . London : Falmer Press .

Gadotti (2010) Gadotti , M. ( 2010 ). Reorienting education practices towards sustainability . Journal of Education for Sustainable Development , 4 ( 2 ), 203 – 211 .

Gedžūne (2013a) Gedžūne , G. ( 2013a ). ICT-supported educational action research in teacher education for sustainability: Interplay of multiple voices in discourse of exclusion . In U. Harkonen (Ed.). Reorientation of teacher education towards sustainability through theory and practice (pp. 333 – 362 ). Finland : University of Eastern Finland .

Gedžūne (2013b) Gedžūne , I. ( 2013b ). Towards inclusive environment ethic in teacher education for sustainability: Insights from an educational action research cycle . In U. Harkonen (Ed.)., Reorientation of teacher education towards sustainability through theory and practice (pp. 109–129, 333–362). Finland : University of Eeastern Finland .

Gedžūne (2014) Gedžūne , G. ( 2014 ). Pre-service teachers’ frames of reference for addressing children’s social exclusion in the classroom . Summary of the Ph.D. theses, Daugavpils University, Saule.

Gedzūne & Gedžūne (2012) Gedžūne , G. , & Gedžūne , I. ( 2012 ). Making sense of inclusion and exclusion through educational action research for sustainability in teacher education . Procedia - Social and Behavioral Sciences , 46 , 3097 – 3101 .

Gedžūne & Gedžūne (2013) Gedžūne , I. , & Gedžūne , G. ( 2013 ). Educational action research to initiate discourse on inclusion in an e-learning environment in teacher education . Educational Action Research , 21 ( 1 ), 72 – 89 . doi: 10.1080/09650792.2013.763419

Gedžūne, Gedžūne, Salīte, & Iliško (2011) Gedžūne , G. , Gedžūne , I. , Salīte , I. , & Iliško , Dz. ( 2011 ). Exploring pre-service teachers’ frames of reference and their orientation towards inclusion or exclusion: Educational action research journey. Proceedings of the 9th JTEFS/BBCC Conference “Sustainable Development, Culture, Education: BBCC Mission - Reorientation of Teacher Education and Research in Education for Sustainable Development ”, Lithuania (pp. 80–102).

Gharajedaghi (2011) Gharajedaghi , J. ( 2011 ). Systems thinking: Managing chaos and complexity. A platform for designing business architecture . Burlington, MA : Elsevier .

Gibson (2004) Gibson , W. E. ( 2004 ). Eco-justice: Unfinished journey . New York, NY : State University of New York Press .

Gilbert (2011) Gilbert , R. ( 2011 ). Professional learning flagship program: Leading curriculum change . Melbourne: The Australian Institute for Teaching and School Leadership.

Giroux & MacLaren (1986) Giroux , H. , & MacLaren , P. ( 1986 ). Teacher education and the politics of engagement: The case for democratic schooling . Harvard Educational Review , 56 ( 3 ), 213 – 238 .

Goodlad (2004) Goodlad , J. ( 2004 ). A place called school . New York , NY: McGraw Hill .

Guseva (2012) Guseva , V. ( 2012 ). Differentiated learning for securing inclusive approach at the first stage of basic education . Summary of the Ph.D. theses, Daugavpils University, Saule.

Guttal (2016) Guttal , S. ( 2016 ). Interrogating the relevance of the Global North-South Divide. Focus on the Global South . Retrieved from https://www.cetri.be/IMG/pdf/

Hargreaves (1994) Hargreaves , A. ( 1994 ). Changing teachers, changing times: Teachers’ work and culture in the postmodern age . New York, NY : Teachers College Press .

Harvey & Stensker (2008) Harvey , L. , & Stensker , B. ( 2008 ). Quality culture: Understandings, boundaries, linkages . European Journal of Education , 43 , 427 – 442 .

Hatcher, Bartlett, Marshall, & Marshall (2009) Hatcher , A. , Bartlett , C. , Marshall , A. , & Marshall , M. ( 2009 ). Two-eyed seeing in the classroom environment: Concepts, approaches, and challenges . Canadian Journal of Science, Mathematics, and Technology Education , 9 ( 3 ), 141 – 153 .

Hauss (2015) Hauss , C. ( 2015 ). Security. 2.0: Dealing with global wicked problems . New York, NY : Rowman & Littlefield .

Heinämäki (2009) Heinämäki , L. ( 2009 ). Protecting the rights of indigenous peoples: Promoting the sustainability of the global environment? International Community Law Review , 11 , 3 – 68 .

Hoffmann & Adams (2005) Hoffmann , A. M. , & Adams , A. B. ( 2005 ). The human capability approach and education for sustainable development: Making the abstract real. Paper presented at the 5th Conference on the Capability Approach, Paris.

Kapyrka & Dockstator (2012) Kapyrka , J. , & Dockstator , M. ( 2012 ). Indigenous knowledges and western knowledges in environmental education: Acknowledging the tensions for the benefits of a “two-worlds” approach . Canadian Journal of Environmental Education , 17 , 97 – 112 .

Kawagley & Barnhardt (1998) Kawagley , A. O. , & Barnhardt , R. ( 1998 ). Education indigenous to place: Western science meets native reality. Retrieved from https://eric.ed.gov/?id=ED426823

Kimmerer (2002) Kimmerer , R. W. ( 2002 ). Weaving traditional ecological knowledge into biological education: A call to action . Bioscience , 52 ( 5 ), 432 – 438 .

Koller & Davidson (2008) Koller , V. , & Davidson , P. ( 2008 ). Social exclusion as a conceptual metaphor: A cross-genre study of British policy making . Discourse and Sociology , 19 ( 3 ), 307 – 331 .

Koopmans & Stamovlasis (2016) Koopmans , M. , & Stamovlasis , D. ( 2016 ). Complex dynamical systems in education. Concepts, methods and applications . Heidelberg : Springer .

Koppenjan & Klijn (2004) Koppenjan , J. , & Klijn , E. H. ( 2004 ). Managing uncertainties and networks. A network approach to problem solving and decision making . London : Routledge .

Kulnieks, Longboat, & Young (2013) Kulnieks , A. , Longboat , D. R. , & Young , K. ( 2013 ). Eco-literacy development through a framework for indigenous and environmental education leadership . Canadian Journal of Environmental Education , 18 , 111 – 125 .

Iliško (2016a) Iliško , Dz . ( 2016a ). A hopeful vision for a just world: Ecofeminist agenda . Journal of the European Society of Women in Theological Research , 24 , 121 – 132 . doi: 10.20142/ESWTR24.0.3170029

Iliško (2016b) Iliško , Dz. ( 2016b ). Inquiry-based educational course in higher education towards sustainable communities: A case study . In W. Leal Filho & P. Pace (Eds.), Teaching education for sustainable development at university level (pp. 125 – 145 ). The Netherlands : Springer . doi: 10.1007/978-3-319-32928-4_9

Iliško, Badjanova, & Ignatjeva (2017) Iliško , Dz ., Badjanova , J. , & Ignatjeva , S. ( 2017 ). Sustainability and unsustainability aspects of social adaptation of children from returning immigrant families. Proceedings of the 10th Annual International Conference of Education, Research and Innovation (pp. 4054–4060). Retrieved from https://doi.org/10.21125/iceri.2017.107

Iliško & Ignatjeva (2014) Iliško , Dz. , & Ignatjeva , S. ( 2014 ). A system approach towards inclusion in schools of Latvia. Proceedings of the 6th International Conference on Education and New Learning Technology , Spain (pp. 3164–3170).

Incheon Declaration and Framework for Action for the Implementation of Sustainable Development Goal 4 Incheon Declaration and Framework for Action for the Implementation of Sustainable Development Goal 4 . Retrieved from http://uis.unesco.org/sites/default/files/documents/education-2030-incheon-framework-for-action-implementation-of-sdg4-2016-en_2.pdf

Inter-agency Expert Group on SDG Indicators (IAEG-SDGs) (2015) Inter-agency Expert Group on SDG Indicators (IAEG-SDGs) . ( 2015 ). Retrieved from https://unstats.un.org/sdgs/iaeg-sdgs/report-iaeg-sdgs/

Kesälahti & Väyrymen (2013) Kesälahti , E. , & Väyrymen , S. ( 2013 ). Learning from our neighbors: Inclusive education in the making a school for all - Development for inclusive education . Povaniemi, Finland : University of Lapland Printing Center .

Kļaviņš & Pelena (2010) Kl¸avin¸š , M. , & Pelena , M. ( 2010 ). Concepts and approaches for implementation of education for sustainable development in the curricula of universities in Latvia . Journal of Baltic Science Education , 9 ( 4 ), 264 – 272 .

Laurie, Nonoyama-Tarumi, McKeown, & Hopkins (2016) Laurie , R. , Nonoyama-Tarumi , Y. , McKeown , R. , & Hopkins , C. ( 2016 ). Contribution of Education for Sustainable Development (ESD) to quality education: A synthesis of research . Journal of Education for Sustainable Development , 10 ( 2 ), 226 – 242 .

Lederman (2007) Lederman , N. G. ( 2007 ). Nature of science: Past, present, and future . In S. K. Abell & N. G. Lederman (Eds.), Handbook of research on science education (pp. 831 – 880 ). Mahwah, NJ : Lawrence Erlbaum Associates .

Lewin (1999) Lewin , R. ( 1999 ). Complexity: Life to the edge of chaos . Chicago, IL : The University of Chicago Press .

Mackenzie, Cologon, & Fenech (2016) Mackenzie , M. , Cologon , K. , & Fenech , M. ( 2016 ). Approaching the inclusive early childhood education of a child labelled with autism from societal relational understanding of disability . Australian Journal of Early Childhood , 41 ( 1 ), 4 – 12 .

Maila & Loubser (2003) Maila , M. W. , & Loubser , C. P. ( 2003 ). Emancipatory indigenous knowledge systems: implications for environmental education in South Africa . South African Journal of Education , 23 ( 4 ), 276 – 280 .

McGuire, Scott, & Shaw (2006) McGuire , J. M. , Scott , S. S. , & Shaw , S. F. ( 2006 ). Universal design and its application in educational environments . Remedial and Special Education , 27 ( 3 ), 166 – 175 .

McKeown (2002) McKeown , R. ( 2002 ). Education for sustainable development Toolkit . Retrieved from http://www.esdtoolkit.org/esd_toolkit_v2.pdf

Menzies & Butler (2007) Menzies , C. R. , & Butler , C. F. ( 2007 ). Returning to selective fishing through indigenous fisheries knowledge: The example of K’moda, Gitxaala territory . The American Indian Quarterly , 31 ( 3 ), 441 – 464 .

Miller (1999) Miller , E. ( 1999 ). Teacher development in the Caribbean . Retrieved from https://www.oas.org/cotep/GetAttach.aspx?lang=es&cId=656&aid=981

Ministry of Education (2012) Ministry of Education . ( 2012 ). National education strategic plan: 2011-2020 . Kingston : Ministry of Education .

Ministry of Education (2015) Ministry of Education . ( 2015 ). Ministry Launches Grade Four TV Programme. Retrieved from http://www.moe.gov.jm/ministry-launches-grade-four-tests-tv-programme

Moreno (2006) Moreno , J. ( 2006 ). The dynamics of curriculum design and development: Scenarios for curriculum evolution . In A. Benavot , C. Braslavsky , & N. Truong (Eds.), School knowledge in comparative and historical perspective: Changing curricula in primary and secondary education (pp. 195 – 209 ). Hong Kong : Comparative Education Research Centre .

Morin (2008) Morin , E. ( 2008 ). On complexity . New York, NY : Hamton Press .

Murdoch & Clark (1994) Murdoch , J. , & Clark , J. ( 1994 ). Sustainable knowledge . Geoforum , 25 ( 2 ), 115 – 132 .

Nandini & Taj (2014) Nandini , N. , & Taj , H. ( 2014 ). Inclusive education: Key role of teachers for its success . International Journal of Informative & Futuristic Research , 1 ( 9 ), 201 – 208 .

National Environmental Education Committee (NEEC) (1998) National Environmental Education Committee (NEEC) . ( 1998 ). The national environmental education action plan for sustainable development . Kingston : NEEC Secretariat .

Nikel & Lowe (2010) Nikel , J. , & Lowe , J. ( 2010 ). Talking of fabrics: A multi-dimensional model of quality in education . Compare , 40 ( 5 ), 589 – 605 .

Nind (2014) Nind , M. ( 2014 ). What is inclusive research? London : Bloomsbury Academic .

Norman (2011) Norman , A. D. ( 2011 ). Living with complexity . London : The MIT Press .

Norwich (2013) Norwich , B. ( 2013 ). Addressing tensions and dilemmas in inclusive education . London : Routledge .

Nussbaum (2006) Nussbaum , M. C. ( 2006 ). Frontiers of justice: Disability, nationality, species membership . Cambridge, MA : The Belknap Press of Harvard University Press .

Nussbaum (2011) Nussbaum , M. C. ( 2011 ). Creating capabilities. The human development approach . Cambridge : Harvard University Press .

OECD (2016a) OECD . ( 2016a ). Better policies for 2030: An OECD action plan on the sustainable development goals .

OECD (2016b) OECD . ( 2016b ). Education in Latvia . Retrieved from http://www.oecd.org/newsroom/latvia-should-continue-improving-quality-of-education-and-focus-more-on-equity.htm

OECD (2017b) OECD . ( 2017b ). Education at a glance 2017 OECD indicators . Paris, France : OECD Publishing . Retrieved from http://dx.doi.org/10.1787/eag-2017-en

Ornstein & Hunkins (2009) Ornstein , A. , & Hunkins , F. ( 2009 ). Curriculum foundations, principles and issues . Boston, MA : Allyn & Bacon .

Overseas Development Institute (2015a) Overseas Development Institute . ( 2015a ). Projecting progress: Reaching the SDGs by 2030 . Retrieved from http://developmentprogress.odi.org/sdgs-scorecard/scorecard_report.pdf

Overseas Development Institute (2015b) Overseas Development Institute . ( 2015b ). Annex to projecting progress: Reaching the SDGs by 2030 . Retrieved from https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/9937.pdf

Pacific Policy Research Centre (2010) Pacific Policy Research Centre . ( 2010 ). 21st Century skills for students and teachers . Research and Evaluation , 1 – 25 . Retrieved from http://www.ksbe.edu/_assets/spi/pdfs/21_century_skills_full.pdf

Planning Institute of Jamaica (PIOJ) (2009) Planning Institute of Jamaica (PIOJ) . ( 2009 ). Vision 2030 Jamaica: National development plan .

PIOJ (2012) Planning Institute of Jamaica ( PIOJ ). ( 2012 ). Analysis of the global competitiveness report 2011-2012. Retrieved from http://www.pioj.gov.jm

Rajiv (2009) Rajiv , M. ( 2009 ). Swami Ranganathananda reader . New Delhi : Rupa & Co .

Rasmussen & Bayer (2014) Rasmussen , J. , & Bayer , M. ( 2014 ). Comparative study of teaching content in teacher education programmes in Canada, Denmark, Finland and Singapore . Journal of Curriculum Studies , 46 ( 6 ), 798 – 818 . doi: 10.1080/00220272.2014.927530

Robertson (2013) Robertson , E. ( 2013 ). The epistemic value of diversity . Journal of Philosophy of Education , 47 , 299 – 310 .

Sachs (2015) Sachs , J. ( 2015 ). The age of sustainable development . New York, NY : Columbia University Press .

Sachs, Schmidt-Traub, Kroll, Durand-Delacre, & Teksoz (2017) Sachs , J. , Schmidt-Traub , G. , Kroll , C. , Durand-Delacre , D. , & Teksoz , K. ( 2017 ). SDG index and dashboards report 2017 . New York, NY: Bertelsmann Stiftung and Sustainable Development Solutions Network (SDSN).

Salīte, Gedžūne, & Gedžūne (2009) Salīte , I. , Gedžūne , G. , & Gedžūne , I. ( 2009 ). Educational action research for sustainability: Seeking wisdom of insight in teacher education . Journal of Teacher Education for Sustainability , 11 ( 2 ), 14 – 30 .

Salter & Tapper (2000) Salter , B. , & Tapper , T. ( 2000 ). The politics of governance in higher education: The case of quality insurance . Political Studies , 48 , 66 – 87 . doi: 10.1007/978-1-4-20-5553-9_10

Slade (2017) Slade , S. ( 2017 , February 22). What do we mean by a quality education? The Blog Retrieved from https://www.huffingtonpost.com/sean-slade/what-do-we-mean-by-a-qual_b_9284130.html

Snively & Corsiglia (1998) Snively , G. , & Corsiglia , J. ( 1998 ). Haida Gwaii field school in culture and education, Queen Charlotte islands. Proceedings of a Faculty Conference , British Columbia.

Spratt & Florian (2015) Spratt , J. , & Florian , L. ( 2015 ). Inclusive pedagogy: From learning to action. Supporting each individual in the context of ‘everybody’ . Teaching and Teacher Education , 49 , 89 – 96 .

Sustainable Development Solution Network (2015) Sustainable Development Solution Network . ( 2015 ). Indicators and a monitoring framework for the sustainable development goals: Launching a data revolution for the SDGs: A report to the Secretary-General of the United Nations by the Leadership Council of the Sustainable Development Solutions Network . Retrieved from http://unsdsn.org/wp-content/uploads/2015/05/FINAL-SDSN-Indicator-Report-WEB.pdf

Taylor & Buttel (1992) Taylor , P. J. , & Buttel , F. H. ( 1992 ). How do we know we have global environmental problems? Science and the globalization of environmental discourse . Geoforum , 23 ( 3 ), 405 – 416 .

Terzi (2014) Terzi , L. ( 2014 ). Refraining inclusive education: Educational equality as capability equality . Cambridge Journal of Education , 44 ( 4 ), 479 – 493 .

Tilbury (2011) Tilbury , D. ( 2011 ). Education for sustainable development: An expert review on processes and learning . Paris, France : UNESCO . Retrieved from unesdoc.unesco.org/images/0019/001914/191442e.pdf

UNESCO (1991) UNESCO . ( 1991 ). World Education Report . Paris : UNESCO . Retrieved from: http://unesdoc.unesco.org/images/0008/000895/089546eo.pd

UNESCO (2000) UNESCO . ( 2000 ). World Education Forum: Final report . Paris : UNESCO . Retrieved from http://unesdoc.unesco.org/images/0012/001211/121117e.pdf

UNESCO (2009) UNESCO . ( 2009 ). Bonn declaration . Paris : UNESCO . Retrieved from http://unesdoc.unesco.org/images/0018/001887/188799e.pdf

UNESCO (2011) UNESCO . ( 2011 ). UNESCO and “Everyone has the right to education” . France : UNESCO . Retrieved from http://unesdoc.unesco.org/images/0021/002127/212715e.pdf

UNESCO (2016) UNESCO . ( 2016 ). Global education monitoring report. Education for people and planet: Creating sustainable futures for all . Paris : UNESCO . Retrieved from http://unesdoc.unesco.org/images/0024/002457/245752e.pdf

UNESCO Institute for Statistics (UIS) (2016) UNESCO Institute for Statistics (UIS) . ( 2016 ). Sustainable development data digest: Laying the foundation to measure Sustainable Development Goal 4 . Montreal : UNESCO Institute for Statistics . Retrieved from http://www.ungei.org/uis-sdg4-digest-2016.PDF

United Nations (UN) (1992a) United Nations (UN) . ( 1992a ). Agenda 21: Programme of action for sustainable development earth summit . Retrieved from https://sustainabledevelopment.un.org/content/documents/Agenda21.pdf

UN (1992b) United Nations (UN) . ( 1992b ). Convention on biological diversity . Retrieved from https://www.cbd.int/doc/legal/cbd-en.pdf

United Nations (2000) United Nations . ( 2000 ). United Nations millennium declaration . Retrieved from http://www.un.org/millennium/declaration/ares552e.htm/

UN (2007) United Nations (UN) . ( 2007 ). United nations declaration on the rights of indigenous peoples 61/295 . Retrieved from www.un.org/esa/socdev/unpfii/documents/DRIPS_en.pdf

UN (2015) United Nations (UN) . ( 2015 , September). Transforming our world: The 2030 agenda for sustainable Development . Retrieved from http://www.un.org/ga/search/view_doc.asp?symbol=A/RES/70/1&Lang=E

UN Conference on Environment and Development (UNCED) (1992a) UN Conference on Environment and Development (UNCED) . ( 1992a ). Agenda 21: Programme of action for sustainable development . New York, NY: United Nations. Retrieved from https://sustainabledevelopment.un.org/content/documents/Agenda21.pdf

UN Conference on Environment and Development (UNCED) (1992b) UN Conference on Environment and Development (UNCED) . ( 1992b ). The Rio declaration on environment and development . United Nations. Retrieved from http://www.unesco.org/education/pdf/RIO_E.PDF

UNESCO (1990) UNESCO . ( 1990 ). World declaration on education for all and framework for action to meet basic learning needs. International Consultative Forum on Education for All . Paris : UNESCO .

UNESCO. (1994). UNESCO . ( 1994 ). The Salamanca statement and framework for action on special needs education . Retrieved from http://www.unesco.org/education/pdf/SALAMA_E.PDF

UNESCO (2000) UNESCO . ( 2000 ). The Dakar Framework for Action . Paris, France : UNESCO .

UNESCO (2002) UNESCO . ( 2002 ). Education for sustainability: From Rio to Johannesburg lessons from a decade of commitment. Paris: UNESCO. Retrieved from http://www.unesdoc.unesco.org/images/0012/001271/127100e.pdf

UNESCO (2005a) UNESCO . ( 2005a ). Contributing to a more sustainable future: Quality education, life skills and education for sustainable development . Paris, France : UNESCO . Retrieved from http://unesdoc.unesco.org/images/0014/001410/141019e.pdf

UNESCO (2005b) UNESCO . ( 2005b ). Guidelines for inclusion. Ensuring access to education for all . Retrieved from http://unesdoc.unesco.org/images/0014/001402/140224e.pdf

UNESCO (2005c) UNESCO . ( 2005c ). UN Decade for Sustainable Development 2005–2014 Draft international implementation scheme . Paris: UNESCO.

UNESCO (2012) UNESCO . ( 2012 ). The education for sustainable development Sourcebook. Education for sustainable development in action learning and training tools . No 4. Paris, France : UNESCO . Retrieved from http://unesdoc.unesco.org/images/0021/002154/215431e.pdf

UNESCO (2014a) UNESCO . ( 2014a ). Aichi-Nagoya Declaration on ESD . Paris, France : UNESCO . Retrieved from http://www.unesco.org/new/fileadmin/MULTIMEDIA/HQ/ERI/pdf/Aichi-Nagoya_Declaration_EN.pdf

UNESCO (2014b) UNESCO . ( 2014b ). Building a better fairer world for the 21st century . Retrieved from http://unesdoc.unesco.org/images/0021/002166/216673E.pdf

UNESCO (2014c) UNESCO . ( 2014c ). UNESCO roadmap for implementing the global action programme on education for sustainable development . Paris, France : UNESCO .

UNESCO (2015a) UNESCO . ( 2015a ). Education 2030 - Incheon Declaration - Towards inclusive and equitable quality education and lifelong learning for all . Paris, France : UNESCO . Retrieved from www.uis.unesco.org/Education/Documents/incheon-framework-for-action-en.pdf

UNESCO (2015b) UNESCO . ( 2015b ). Repositioning curriculum in education quality & development-relevance . International Bureau of Education . Retrieved from http://www.ibe.unesco.org

UNESCO (2015c) UNESCO . ( 2015c ). Rethinking education: Towards a global common good? Paris, France : UNESCO . Retrieved from http://unesdoc.unesco.org/images/0023/002325/232555e.pdf

UNESCO (2017a) UNESCO . ( 2017a ). Unpacking sustainable development goal 4: Education 2030 . Retrieved from http://www.right-to-education.org/sites/right-to-education.org/files/resource-attachments/Unesco_Guide_to_unpacking_SDG4_2015_En.pdf

UNESCO (2017b) UNESCO . ( 2017b ). Global education monitoring report 2017/18 - accountability in education: Meeting our commitments . Paris, France: UNESCO.

UN (2017). UN Economic and Social Council (ECOSOC). ( 2017 ). Economic commission for Europe, conference of European statisticians’ road map on statistics for sustainable development goals. New York and Geneva, United Nations.

UNESCO-IBE (2008) UNESCO-IBE . ( 2008 ). Conclusions and recommendations of the 48th session of the International Conference on Education (ED/BIE/CONFINED48/5). Geneva, Switzerland: UNESCOIBE. Retrieved from http://www.ibe.unesco.org/fileadmin/user_upload/Policy_Dialogue/48th_ICE/ICE_FINAL_REPORT_eng.pdf

UNESCO (2017a) UNESCO Institute for Statistics (UIS). (2017a). The Investment Case for SDG4 Data, Concept Note, Technical Cooperation Group on SDG4Education 2030 Indicators. Dubai, United Arab Emirates.

UNESCO (2017b) UNESCO Institute for Statistics (UIS). (2017b). SDG4 data digest 2017 - the quality factor: Strengthening national data to monitor sustainable development goal4. Montreal: UNESCO Institute for Statistics.

UN High Level PoliticalForum (HLPF) (2017a) UN High Level PoliticalForum (HLPF) . ( 2017a ). Leaving no-one behind: what do we mean and what does it entail? Education and Academia Stakeholder Group. Retrieved from https://sustainabledevelopment.un.org/content/documents/10110Education%20and%20Academia%20Stakeholder%20Group_FINAL_CLEAN.pdf

UN-HLPF (2017b) UN High Level PoliticalForum (HLPF) . ( 2017b ). Voluntary National Reviews Synthesis Report, Division for Sustainable Development Department of Economic and Social Affairs, United Nations High-Level Political Forum on Sustainable Development. Retrieved from https://sustainabledevelopment.un.org/

UN-HLPF (2018) UN High Level PoliticalForum (HLPF) . ( 2018 ). Voluntary National Reviews Database Guidance, Division for Sustainable Development, Department of Social and Economic Affairs. Retrieved from https://sustainabledevelopment.un.org/vnrs/

UNICEF (2016) UNICEF . ( 2016 ). Promoting quality education . Retrieved from http://www.unicef.org

United Nations Secretary General (UNSG) (2014) United Nations Secretary General (UNSG) . ( 2014 ). The road to dignity by 2030: Ending poverty, transforming all lives and protecting the planet. Synthesis report of the Secretary-General on the post-2015 agenda . New York, NY: UN.

Waitoller & Artiles (2013) Waitoller , F. R. , & Artiles , A. J. ( 2013 ). A decade of professional development research for inclusive education: A critical review and notes for a research program . Review of Educational Research , 83 ( 3 ), 319 – 356 .

Watkins & Donnelly (2012) Watkins , A. , & Donnelly , V. ( 2012 ). Teacher education for inclusion in Europe: Challenges and opportunities . In C. Forlin (Ed.). Future directions for inclusive teacher education. An international perspective (pp. 192 – 202 ). Abingdon : Routledge .

Whyte (2013) Whyte , K. P. ( 2013 ). On the role of traditional ecological knowledge as a collaborative concept: A philosophical study . Ecological Processes , 2 ( 7 ). Retrieved from http://ecologicalprocesses.springeropen.com/articles/10.1186/2192-1709-2-7

Wittgenstein Centre for Demography and Global Human Capital (2015) Wittgenstein Centre for Demography and Global Human Capital . ( 2015 ). Wittgenstein Center Data Explorer . Vienna, Austria : Wittgenstein Centre for Demography and Global Human Capital .

World Bank (2017) World Bank . ( 2017 ). Jamaica Performance and learning review of the country partnership strategy FY14-17 (English) . Washington, DC : World Bank Group . Retrieved from http://documents.worldbank.org/curated/en/655091503334217656/Jamaica-Performance-and-learning-review-of-the-country-partnership-strategy-FY14-17

World Commission on Environment and Development (WCED) (1987) World Commission on Environment and Development (WCED) . ( 1987 ). Our common future . Oxford : Oxford University Press .

Zacharuk (2011) Zacharuk . ( 2011 ). Inclusive education – The chance for all pupils . Meritum , 1 ( 20 ), 2 – 7 .

Book Chapters

All feedback is valuable.

Please share your general feedback

Report an issue or find answers to frequently asked questions

Contact Customer Support

Discipline-Based Education Research: Understanding and Improving Learning in Undergraduate Science and Engineering (2012)

Chapter: references.

American Association for the Advancement of Science. (2011). Vision and change in undergraduate biology education: A call to action. Washington, DC: American Association for the Advancement of Science.

Association of American Universities. (2011). Five-year initiative for improving undergraduate STEM education: Discussion draft. Washington, DC: Association of American Universities.

Bodner, G. (2011). Status , contributions , and future directions of discipline-based education research: The development of research in chemical education as a field of study . Paper presented at the Second Committee Meeting on the Status, Contributions, and Future Directions of Discipline-Based Education Research. Available: http://www7.nationalacademies.org/bose/DBER_Bodner_October_Paper.pdf .

Boyer, E.L. (1990). Scholarship reconsidered: Priorities of the professoriate . San Francisco, CA: Jossey-Bass.

Brewer, C., and Smith, D. (Eds.). (2011). Vision and change in undergraduate biology education: A call to actio n. Available: http://visionandchange.org/finalreport .

Cooper, M., Grove N., Underwood, S., and Klymkowsky, M. (2010). Lost in Lewis structures: An investigation of student difficulties in developing representational competence. Journal of Chemical Education , 87 (8), 869-874.

Friedenberg, J., and Silverman, G. (2006). Cognitive science: An introduction to the study of the mind. Thousand Oaks, CA: Sage.

Hatch, T. (2005). Into the classroom: Developing the scholarship of teaching and learning . San Francisco, CA: Jossey-Bass.

Huber, M., and Hutchings, P. (2005). The advancement of learning: Building the teaching commons . San Francisco, CA: Jossey-Bass/Carnegie Foundation for the Advancement of Teaching.

Hutchings, P., Huber, M., and Ciccone, A. (2011). The scholarship of teaching and learning reconsidered: Institutional integration and impact . San Francisco, CA: Jossey-Bass.

Mayer, R.E. (2011). Applying the science of learning to undergraduate science education . Paper presented at the Third Committee Meeting on Status, Contributions, and Future Directions of Discipline-Based Education Research. Available: http://www7.nationalacademies.org/bose/DBER_Mayer_December_Paper.pdf .

National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. (2011). Expanding underrepresented minority participation : America’s science and technology talent at the crossroads. Committee on Underrepresented Groups and the Expansion of the Science and Engineering Workforce Pipeline.Committee on Science, Engineering, and Public Policy and Policy and Global Affairs. Washington, DC: The National Academies Press.

National Research Council. (2002). Scientific research in education. Committee on Scientific Principles for Education Research, R.J. Shavelson and L. Towne, Eds. Committee on Scientific Principles for Education Research, Center for Education. Division of Behavioral and Social Sciences and Education. Washington, DC: National Academy Press.

National Research Council. (2012). A framework for K-12 science education practices , crosscutting concepts , and core ideas . Committee on a Conceptual Framework for New K-12 Science Education Standard, Board on Science Education. Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.

Novick, L.R., and Catley, K.M. (in press). Reasoning about evolution’s grand patterns: College students’ understanding of the tree of life. American Educational Research Journal .

President’s Council of Advisors on Science and Technology. (2012). Engage to excel: Producing one million additional college graduates with degrees in science, technology, engineering, and mathematics . Washington, DC: Executive Office of the President, President’s Council of Advisors on Science and Technology.

Project Kaleidoscope. (2011a). Volume IV: What works , what matters , what lasts: 2004-present: PKAL’s online publication . Washington, DC: Project Kaleidoscope. Available: http://www.pkal.org/collections/VolumeIV.cfm .

Project Kaleidoscope. (2011b). What works in facilitating interdisciplinary learning in science and mathematics . Washington, DC: Association of American Colleges and Universities.

Stokes, D.E. (1997). Pasteur’s quadrant: Basic science and technological innovation . Washington, DC: Brookings Institution Press.

ABET. (2009). ABET criteria for evaluating engineering programs . Baltimore, MD: ABET.

Bailey, J.M. (2011). Astronomy education research: Developmental history of the field and summary of the literature . Paper presented at the Second Committee Meeting on the Status, Contributions, and Future Directions of Discipline-Based Education Research. Available: http://www7.nationalacademies.org/bose/DBER_Bailey_Commissioned%20Paper.pdf .

Bauer, C.F., Clevenger, J.V., Cole, R.S., Jones, L.L., Kelter, P.B., Oliver-Hoyo, M.T., and Sawrey, B.A. (2008). Association report, ACS Division of Chemical Education: Hiring and promotion in chemical education. Journal of Chemical Education , 85 , 898-901.

Beichner, R. (2009). An introduction to physics education research. In C. Henderson and K. Harper (Eds.), Getting started in PER. College Park, MD: American Association of Physics Teachers. Available: http://www.per-central.org/items/detail.cfm?ID=8806 .

Bloom, B.S. (1956). Taxonomy of educational objectives , the classification of educational goals (1st ed.). New York: Longmans, Green.

Bush, S.D., Pelaez, N.J., Rudd, J.A., Stevens, M.T., Williams, K.S., Allen, D.E., and Tanner, K.D. (2006). On hiring science faculty with education specialties for your science (not education) department. CBE-Life Sciences Education , 5 (4), 297-305.

Bush, S.D., Pelaez, N.J., Rudd, J.A., Stevens, M.T., Tanner, K.D., and Williams, K.S. (2008). The pipeline: Science faculty with education specialties. Science , 322 (5909), 1795-1796.

Bush, S.D., Pelaez, N.J., Rudd, J.A., Stevens, M.T., Tanner, K.D., and Williams, K.S. (2011). Investigation of science faculty with education specialties within the largest university system in the United States. CBE-Life Sciences Education , 10 (1), 25-42.

Clary, R.M., Brzuszek, R.F., and Wandersee, J.H. (2009). Students’ geocognition of deep time, conceptualized in an informal educational setting. Journal of Geoscience Education , 57 (4), 275-285.

Cummings, K. (2011). A developmental history of physics education research . Paper presented at the Second Committee Meeting on the Status, Contributions, and Future Directions of Discipline-Based Education Research. Available: http://www7.nationalacademies.org/bose/DBER_Cummings_October_Paper.pdf .

D’Avanzo, C. (2008). Biology concept inventories: Overview, status, and next steps. Bioscience , 58 (11), 1079-1085.

DeHaan, R.L. (2011). Education research in the biological sciences: A nine-decade review . Paper presented at the Second Committee Meeting on the Status, Contributions, and Future Directions of Discipline-Based Education Research. Available: http://www7.nationalacademies.org/bose/DBER_DeHaan_October_Paper.pdf .

Dirks, C. (2011). The current status and future direction of biology education research . Paper presented at the Second Committee Meeting on the Status, Contributions, and Future Directions of Discipline-Based Education Research. Available: http://www7.nationalacademies.org/bose/DBER_Dirks_October_Paper.pdf .

Docktor, J.L., and Mestre, J.P. (2011). A synthesis of discipline-based education research in physics . Paper presented at the Second Committee Meeting on the Status, Contributions, and Future Directions of Discipline-Based Education Research. Available: http://www7.nationalacademies.org/bose/DBER_Docktor_October_Paper.pdf .

Dodick, J., and Orion, N. (2003). Cognitive factors affecting student understanding of geologic time. Journal of Research in Science Teaching , 40 (4), 415-442.

Fensham, P.J. (2004). Defining an identity: The evolution of science education as a field of research . Boston, MA: Springer.

Feynman, R.P., Leighton, R.B., and Sands, M. (1964). The Feynman lectures on physics . Boston, MA: Addison-Wesley.

Finlay, G.C. (1962). The Physical Science Study Committee. The School Review, 70 (1), 63-81.

French, A.P. (1968). Special relativity . New York: Norton.

Hestenes, D., Wells, M., and Swackhamer, G. (1992). Force concept inventory. The Physics Teacher, 30 , 141-158.

Holton, G. (2003). The Project Physics Course, then and now. Cambridge, MA: Harvard University.

Irwin, L. (1970). A brief historical account and introduction to the Earth Science Curriculum Project. The High School Journal, 53 (4), 241-249.

Kali, Y., and Orion, N. (1996). Spatial abilities of high-school students in the perception of geologic structures. Journal of Research in Science Teaching, 33 , 369-391.

Karplus, R. (1964). The science curriculum improvement study. Journal of Research in Science Teaching, 2 , 293-303.

Kastens, K.A., and Ishikawa, T. (2006). Spatial thinking in the geosciences and cognitive sciences. Geological Society of America Special Papers , 413 , 53-76.

Kern, E.L., and Carpenter, J.R. (1984). Enhancement of student values, interests and attitudes in earth science through a field-oriented approach. Journal of Geological Education , 32 (5), 299-305.

Kern, E.L., and Carpenter, J.R. (1986). Effect of field activities on student learning. Journal of Geological Education , 34 (3), 180-183.

Kittel, C., Knight, W.D., and Ruderman, M.A. (1965). Mechanics: Berkeley physics course: Volume 1 . New York: McGraw-Hill.

Libarkin, J., and Finkelstein, N. (2011). Seeding DBER: Evaluating the early history of the National Science Foundation’s Postdoctoral Fellowships in Science , Mathematics , Engineering and Technology Education (PFSMETE) as a pathway into discipline-based education research . Paper prepared for the Committee on the Status, Contributions, and Future Directions of Discipline-Based Education Research. Available: http://www7.nationalacademies.org/bose/DBER5_Noah_Finkelstein_Julie_Libarkin.pdf .

Lohmann, J., and Froyd, F. (2011). Chronological and ontological development of engineering education as a field of scientific inquiry . Paper presented at the Second Committee Meeting on the Status, Contributions, and Future Directions of Discipline-Based Education Research. Available: http://www7.nationalacademies.org/bose/DBER_Lohmann_Froyd_October_Paper.pdf .

Manduca, C.A, and Mogk, D. (2006). Earth and mind: How geologists think and learn about the earth . Boulder, CO: Geological Society of America.

Manduca, C.A., Mogk, D.W., and Stillings, N. (2004). Bringing research on learning to the geosciences . Northfield, MN: Carleton College, Science Education Resource Center. Available: http://serc.carleton.edu/files/research_on_learning/ROL0304_2004.pdf .

Manduca, C.A., Mogk, D.W., Tewksbury, B., Macdonald, R.H., Fox, S.P., Iverson, E.R., Kirk, K., McDaris, J., Ormand, C., and Bruckner, M. (2010). SPORE: Science prize for online resources in education. On the cutting edge: Teaching help for geoscience faculty. Science , 327 (5969), 1095-1096.

Matthews, M.R. (1994). Science teaching: The role of history and philosophy of science . New York: Routledge.

Meltzer, D., McDermont, L., Heron, P., Redish, E., and Beichner, R. (2004). A call to the AAPT executive board and publications committee to expand publication of physics education research articles within the American Journal of Physics . Available: http://www.ncs u.edu/PER/Articles/CallToAAPT.pdf .

National Academy of Engineering. (2004). The engineer of 2020: Visions of engineering in the new century . Washington, DC: The National Academies Press.

National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. (2005). Facilitating interdisciplinary research . Committee on Facilitating Interdisciplinary Research. Committee on Science, Engineering, and Public Policy. Washington, DC: The National Academies Press.

National Research Council. (2006). Learning to think spatially: GIS as a support system in the K-12 curriculum . Committee on Support for Thinking Spatially: The Incorporation of Geographic Information Science Across the K-12 Curriculum. Geographical Sciences Committee. Board on Earth Sciences and Resources, Division on Earth and Life Studies. Washington, DC: The National Academies Press.

National Science Foundation. (1992). America’s academic future: A report of the presidential young investigator colloquium on U.S. engineering , mathematics , and science education for the year 2010 and beyond . NSF 91-150. Arlington, VA: National Science Foundation.

National Science Foundation. (1997). Geoscience education: A recommended strategy. Report based on August 29-30, 1996, workshop from the Geoscience Education Working Group to the Advisory Committee for Geosciences and the Directorate for Geosciences of the National Science Foundation, Arlington, VA. Available: http://www.nsf.gov/pubs/1997/nsf97171/nsf97171.htm .

Orion, N., and Hofstein, A. (1994). Factors that influence learning during a scientific field trip in a natural environment. Journal of Research in Science Teaching , 31 (10), 1097-1119.

Orion, N., Hofstein, A., Tamir, P., and Giddings, G.J. (1997). Development and validation of an instrument for assessing the learning environment of outdoor science activities. Science Education , 81 (2), 161-171.

Petcovic, H.L., Libarkin, J.C., and Baker, K.M. (2009). An empirical methodology for investigating geocognition in the field. Journal of Geoscience Education , 57 (4), 316-328.

Pfundt, H., and Duit, R. (1988). Bibliography: Students’ alternative frameworks and science education (2nd ed.). Kiel, Germany: Institute for Science Education.

Rhoten, D., and Parker, A. (2004). Risks and rewards of an interdisciplinary research path. Science , 306 (5704), 2046.

Rudolph, F. (1990). The American college and university: A history. Athens: University of Georgia Press.

Rudolph, J.L. (2002). Scientists in the classroom: The Cold War reconstruction of American science education . New York: Palgrave Macmillan.

Semsar, K., Knight, J.K., Birol, G., and Smith, M.K. (2011). The Colorado Learning Attitudes About Science Survey (CLASS) for use in biology. CBE-Life Sciences Education , 10 (3), 268-278.

Strobel, J., Evangelou, D., Streveler, R.A., and Smith, K.A. (2008). The many homes of engineering education research: Historical analysis of PhD dissertations . Paper presented at the Research in Engineering Education Symposium, Davos, Switzerland. Available: http://www.engconfintl.org/8axabstracts/Session%204A/rees08_submission_16.pdf .

Svinicki, M. (2011). Synthesis of the research on teaching and learning in engineering since the implementation of ABET Engineering Criteria 2000 . Paper presented at the Second Committee Meeting on the Status, Contributions, and Future Directions of Discipline-Based Education Research. Available: http://www7.nationalacademies.org/bose/DBER_Svinicki_October_Paper.pdf .

The Steering Committee of the National Engineering Education Research Colloquies. (2006a). The research agenda for the new discipline of engineering education. Journal of Engineering Education , 95 (4), 259-261.

The Steering Committee of the National Engineering Education Research Colloquies. (2006b). The National Engineering Education Research Colloquies. Journal of Engineering Education, 95 (4), 257-258.

Ausubel, D. (2000). The acquisition and retention of knowledge: A cognitive view. Boston, MA: Kluwer Academic.

Bailey, J.M., Slater, S.J., and Slater, T.F. (2011). Conducting astronomy education research: A primer . New York: W.H. Freeman.

Baily, C., and Finkelstein, N.D. (2011). Interpretive themes in quantum physics: Curriculum development and outcomes. AIP Conference Proceedings, 1413 , 107-110.

Bassok, M., and Novick, L.R. (2012). Problem solving. In K.J. Holyoak and R.G. Morrison (Eds.), Oxford handbook of thinking and reasoning (pp. 413-432). New York: Oxford University Press.

Bhattacharyya, G., and Bodner, G.M. (2005). It gets me to the product: How students propose organic mechanisms. Journal of Chemical Education , 82 (9), 1402-1407.

Bretz, S.L., and Nakhleh, M.B. (Eds). (2001). Piaget, constructivism, and beyond. Journal of Chemical Education, 78 (8), 1107.

Brown, J.S., Collins, A., and Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher , 18 (1), 32-42.

Cervato, C., and Frodeman, R. (2012). The significance of geologic time: Cultural, educational, and economic frameworks. Geological Society of America Special Papers, 486 , 19-27.

Dahl, J., Anderson, S.W., and Libarkin, J.C. (2005). Digging into earth science: Alternative conceptions held by K-12 teachers. Journal of Science Education , 12 (2), 65-68.

DeLaughter, J.E., Stein, S., and Bain, K.R. (1998). Preconceptions abound among students in an introductory earth science course. EOS, Transactions of the American Geophysical Union, 79 , 429-432.

Dewey, J. (1916). Democracy and education. New York: Macmillan.

Dodick, J., and Orion, N. (2006). Building an understanding of geological time: A cognitive synthesis of the macro and micro scales of time. Geological Society of America Special Papers , 413 , 77-93.

Gautier, C., Deutsch, K., and Rebich, S. (2006). Misconceptions about the greenhouse effect. Journal of Geoscience Education, 54 , 386-395.

Hansen, R., Long, L., and Dellert, T. (2002). Experiences using flying models in competition and coursework . Proceedings of the American Society for Engineering Education Zone 1 Conference. West Point, NY: American Society for Engineering Education.

Johnstone, A.H. (1991). Why is science difficult to learn? Things are seldom what they seem. Journal of Computer Assisted Learning, 7 , 75-83.

Kastens, K.A., Agrawal, S., and Liben, L.S. (2009). How students and field geologists reason in integrating spatial observations from outcrops to visualize a 3-D geological structure. International Journal of Science Education , 31 (3), 365-393.

Kusnick, J. (2002). Growing pebbles and conceptual prisms: Understanding the source of student misconceptions about rock formation. Journal of Geosciences Education , 50 (1), 31-39.

Lave, J., and Wenger, E. (1991). Situated learning. Legitimate peripheral participation. Cambridge, MA: Cambridge University Press.

Libarkin, J.C., Kurdziel, J.P., and Anderson, S.W. (2007). College student conceptions of geological time and the disconnect between ordering and scale. Journal of Geoscience Education , 55 (5), 413-422.

Liben, L.S., and Titus, S. (2012). The importance of spatial thinking for geoscience education: Insights from the crossroads of geoscience and cognitive science. Geological Society of America Special Papers, 86 , 51-70.

Marx, S.M., Weber, E.U., Orlove, B.S., Leiserowitz, A., Krantz, D.K., Roncali, C., and Phillips, J. (2007). Communication and mental processes: Experiential and analytic processing of uncertain climate information. Global Environmental Change, 17 , 47-58.

Maskall, J., and Stokes, A. (2008). Designing effective fieldwork for the environmental and natural sciences . Plymouth, UK: Higher Education Academy Subject Centre for Geography, Earth and Environmental Sciences.

McGourty, J., Scoles, K., and Thorpe, S.W. (2002). Web-based student evaluation of instruction: Promises and pitfalls . Paper presented at the Forty-second Annual Forum of the Association for Institutional Research, Toronto, Canada. Available: http://web.augsburg.edu/~krajewsk/educause2004/webeval.pdf .

Mogk, D., and Goodwin, C. (2012). Learning in the field: Synthesis of research on thinking and learning in the geosciences. Geological Society of America Special Papers, 486 , 131-163.

Mohan, L., Chen, J., and Anderson, C.W. (2009). Developing a multi-year learning progression for carbon cycling in socio-ecological systems. Journal of Research in Science Teaching , 46 (6), 675-698.

National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. (2005). Facilitating interdisciplinary research. Committee on Facilitating Interdisciplinary Research. Committee on Science, Engineering, and Public Policy. Washington, DC: The National Academies Press.

National Research Council. (2003). Bio2010: Transforming undergraduate education for future research biologists. Committee on Undergraduate Biology Education to Prepare Research Scientists for the 21st Century. Board on Life Sciences, Division on Earth and Life Studies. Washington, DC: The National Academies Press.

National Research Council. (2011). Promising practices in undergraduate science , technology , engineering , and mathematics education: Summary of two workshops. N. Nielsen, Rapporteur. Planning Committee on Evidence on Selected Innovations in Undergraduate STEM Education. Board on Science Education, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.

Orgill, M., and Bodner, G.M. (2006). An analysis of the effectiveness of analogy use in college-level biochemistry textbooks. Journal of Research in Science Teaching , 43 (10), 1040-1060.

Orgill, M., and Bodner, G. (2007). Locks and keys: An analysis of biochemistry students’ use of analogies. Biochemistry and Molecular Biology Education , 35 (4), 244-254.

Pollock, S., Pepper, R., Chasteen, S., and Perkins, K. (2011). Multiple roles of assessment in upper-division physics course reforms. AIP Conference Proceedings, 1413, 307-310.

Rebich, S., and Gautier, C. (2005). Concept mapping to reveal prior knowledge and conceptual change in a mock summit course on global climate change. Journal of Geoscience Education , 53 (4), 355-365.

Sandi-Urena, S., Cooper, M.M., and Stevens, R.H. (2011). Enhancement of metacognition use and awareness by means of a collaborative intervention. International Journal of Science Education , 33 (3), 323-340.

Sanger, M.J. (2008). Using inferential statistics to answer quantitative chemical education research questions. In D. Bunce and R. Cole (Eds.), Nuts and bolts of chemical education research (pp. 101-133). Washington, DC: American Chemical Society and Oxford University Press.

Smith, T.I., Thompson, J.R., and Mountcastle, D.B. (2010). Addressing student difficulties with statistical mechanics: The Boltzmann factor. AIP Conference Proceedings, 1289 , 305-308.

Shepardon, D.P., Wee, B., Priddy, M., Schellenberger, L., and Harbor, J. (2007). What is a watershed? Implications for student conceptions for environmental science education and the National Science Education Standards. Science Education, 91 , 555-577.

Slater, S.J., Slater, T.F., and Bailey, J.M. (2011). Discipline-based education research: A scientist’s guide . New York: W.H. Freeman.

Sterman, J.D., and Sweeney, L.B. (2007). Understanding public complacency about climate change: Adults’ mental models of climate change violate conservation of matter. Climatic Change , 80 (3-4), 213-238.

Stillings, N. (2012). Complex systems in the geosciences and in geoscience learning. Geological Society of America Special Papers , 486 , 97-111.

Sundberg, M.D., Dini, M.L., and Li, E. (1994). Decreasing course content improves student comprehension of science and attitudes towards science in freshman biology. Journal of Research in Science Teaching, 31 , 679-693.

Svinicki, M. (2011). Synthesis of the research on teaching and learning in engineering since the implementation of ABET engineering criteria 2000 . Paper presented at the Second Committee Meeting on the Status, Contributions, and Future Directions of Discipline-Based Education Research. Available: http://www7.nationalacademies.org/bose/DBER_Svinicki_October_Paper.pdf .

Towns, M.H. (2008). Mixed methods designs in chemical education research. In D. Bunce and R. Cole (Eds.), Nuts and bolts of chemical education research (pp. 135-148). Washington, DC: American Chemical Society and Oxford University Press.

Weber, E. (2006). Experience-based and description-based perceptions of long-term risk: Why global warming does not scare us (yet). Climatic Change, 77 , 103-120.

Ausubel, D.P., Novak, J.D., and Hanesian, H. (1978). Educational psychology . New York: Werbel and Peck.

Bailey, J.M., and Slater, T.F. (2005). A contemporary review of K-16 astronomy education research. In O. Engvold (Ed.), Highlights of astronomy (vol. 13, pp. 1029-1031). San Francisco, CA: Astronomical Society of the Pacific.

Bailey, J.M., Johnson, B., Prather, E.E., and Slater, T.F. (2011). Development and validation of the Star Properties Concept Inventory. International Journal of Science Education , 1-30.

Bardar, E.M., Prather, B.K., and Slater, T.F. (2006). Development and validation of the Light and Spectroscopy Concept Inventory. Astronomy Education Review , 5 (2), 103-113.

Barke, H., Hazari, A., and Yitbarek, S. (2009). Misconceptions in chemistry: Addressing perceptions in chemical education. Berlin/Heidelburg: Springer-Verlag.

Besterfield-Sacre, M., Gerchak, J., Lyons, M., Shuman, L.J., and Wolfe, H. (2004). Scoring concept maps: An integrated rubric for assessing engineering education. Journal of Engineering Education , 93 (2), 105-115.

Blackwell, W.H., Powell, M.J., and Dukes, G.H. (2003). The problem of student acceptance of evolution. Journal of Biological Education , 37 (2), 58-67.

Bodner, G.M. (1991). I have found you an argument: The conceptual knowledge of beginning chemistry graduate students. Journal of Chemical Education , 68 (5), 385-388.

Brown, D.E., and Clement, J. (1989). Overcoming misconceptions via analogical reasoning: Abstract transfer versus explanatory model construction. Instructional Science , 18 (4), 237-261.

Cabrera, A.F., Colbeck, C.L., and Terenzini, P.T. (2001). Developing performance indicators for assessing classroom teaching practices and student learning: The case of engineering. Research in Higher Education , 42 (3), 327-352.

Camp, C.W., Clement, J.J., and Brown, D. (1994). Preconceptions in mechanics: Lessons dealing with students’ conceptual difficulties . Dubuque, IA: Kendall/Hunt.

Canpolat, N., Pinarbasi, T., and Sözbilir, M. (2006). Prospective teachers’ misconceptions of vaporization and vapor pressure. Journal of Chemical Education , 83 (8), 1237-1242.

Case, J.M., and Fraser, D.M. (1999). An investigation into chemical engineering students’ understanding of the mole and the use of concrete activities to promote conceptual change. International Journal of Science Education , 21 (12), 1237-1249.

Catley, K.M., and Novick, L.R. (2009). Digging deep: Exploring college students’ knowledge of macroevolutionary time. Journal of Research in Science Teaching , 46 (3), 311-332.

Champagne, A.B., Gunstone, R.F., and Klopfer, L.E. (1985). Effecting changes in cognitive structures among physics students. In L. West and L. Pines (Eds.), Cognitive structure and conceptual change . Orlando, FL: Academic Press.

Cheek, K.A. (2010) A summary and analysis of twenty-seven years of geoscience conceptions research. Journal of Geoscience Education, 58 (3), 122-134.

Chi, M.T.H. (2005). Commonsense conceptions of emergent processes: Why some misconceptions are robust. Journal of the Learning Sciences , 14 (2), 161-199.

Chi, M.T.H. (2008). Three types of conceptual change: Belief revision, mental model transformation, and categorical shift. In S. Vosniadou (Ed.), International handbook of research on conceptual change (pp. 61-82). New York: Routledge.

Chinn, C.A., and Brewer, W.F. (1993). The role of anomalous data in knowledge acquisition: A theoretical framework and implications for science instruction. Review of Educational Research , 63 (1), 1-49.

Clement, J. (1993). Using bridging analogies and anchoring intuitions to deal with students’ preconceptions in physics. Journal of Research in Science Teaching , 30 (10), 1241-1257.

Clement, J. (2008). The role of explanatory models in teaching for conceptual change. In S. Vosniadou (Ed.), International handbook of research on conceptual change (pp. 417-452). New York: Routledge.

Clement, J., Brown, D.E., and Zietsman, A. (1989). Not all preconceptions are misconceptions—finding anchoring conceptions for grounding instruction on students’ intuitions. International Journal of Science Education , 11 (5), 554-565.

Cummings, K., Marx, J., Thornton, R., and Kuhl, D. (1999). Evaluating innovation in studio physics. American Journal of Physics , 67 (Suppl. 1), S38-S44.

diSessa, A.A. (1982). Unlearning Aristotelian physics: A study of knowledge-based learning. Cognitive Science, 6, 37-75.

diSessa, A.A. (1988). Knowledge in pieces. In G. Forman and P. Putall (Eds.), Constructivism in the computer age (pp. 49-70). Hillsdale, NJ: Lawrence Erlbaum Associates.

diSessa, A.A. (2008). A bird’s-eye view of the “pieces” vs. “coherence” controversy (from the “pieces” side of the fence). In S. Vosniadou (Ed.), International handbook of research on conceptual change (pp. 35-60). New York: Routledge.

diSessa, A.A., Gillespie, N., and Esterly, J. (2004). Coherence vs. fragmentation in the development of the concept of force. Cognitive Science , 28 , 843-900.

Englebrecht, A.C., Mintzes, J.J., Brown, L.M., and Kelso, P.R. (2005). Probing understanding in physical geology using concept maps and clinical interviews. Journal of Geoscience Education , 53 (3), 263-270.

Ericsson, K.A., Krampe, R.Th., and Tesch-Römer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100 (3), 363-406.

Gabel, D.L., Samuel, K.V., and Hunn, D. (1987). Understanding the particulate nature of matter. Journal of Chemical Education , 64 (8), 695-697.

Henderleiter, J., Smart, R., Anderson, J., and Elian, O. (2001). How do organic chemistry students understand and apply hydrogen bonding? Journal of Chemical Education , 78 (8), 1126-1130.

Hestenes, D., and Halloun, I. (1995). Interpreting the Force Concept Inventory: A response to March 1995 critique by Huffman and Heller. Physics Teacher , 33 , 504-506.

Hestenes, D., Wells, M., and Swackhamer, G. (1992). Force Concept Inventory. The Physics Teacher , 30 , 141-158.

Heywood, J. (2006). Curriculum change and changing the curriculum. In J. Heywood (Ed.), Engineering education: Research and development in curriculum and instruction (Ch. 7, pp. 177-197). Hoboken, NJ: John Wiley & Sons.

Hidalgo, A., Fernando, S., and Otero, J. 2004. An analysis of the understanding of geological time by students at secondary and post-secondary level. International Journal of Science Education , 26 (7), 845-857.

Hornstein, S., Prather, E., English, T., Desch, S., and Keller, J. (2011). Development and testing of the Solar System Concept Inventory. Bulletin of the American Astronomical Society , 43.

Hubisz, J. (2001). Report on a study of middle school physical science texts. The Physics Teacher , 5 (39), 304-309.

Huffman, D., and Heller, P. (1995). What does the Force Concept Inventory actually measure? Physics Teacher , 33 (2), 138-143.

Jee, B.D., Uttal, D.H., Gentner, D., Manduca, C., Shipley, T.F., Tikoff, B., Ormand, C.J., and Sageman, B. (2010). Commentary: Analogical thinking in geoscience education. Journal of Geoscience Education , 58 (1), 2-13.

Johnson, J.K., and Reynolds, S.J. (2005). Concept sketches—using student and instructor-generated, annotated sketches for learning, teaching, and assessment in geology courses. Journal of Geoscience Education , 53 (1), 85-95.

Johnstone, A.H. (1982). Macro and microchemistry. Chemistry in Britain , 18 (6), 409-410.

Johnstone, A.H. (1991). Why is science difficult to learn? Things are seldom what they seem. Journal of Computer Assisted Learning , 7 , 75-83.

Kastens, K.A., and Manduca, C.A. (2012). Fostering knowledge integration in geoscience education. Geological Society of America Special Papers, 486 , 183-206.

King, C.J.H. (2010). An analysis of misconceptions in science textbooks: Earth science in England and Wales. International Journal of Science Education , 32 (5), 565-601.

Kusnick, J. (2002). Growing pebbles and conceptual prisms: Understanding the source of student misconceptions about rock formation. Journal of Geoscience Education , 50 (1), 31-39.

Libarkin, J. (2008). Concept inventories in higher education science . Paper presented at the National Research Council’s Workshop on Linking Evidence to Promising Practices in STEM Undergraduate Education, Washington, DC. Available: http://www7.nationalacademies.org/bose/Libarkin_CommissionedPaper.pdf .

Lindell, R., and Sommer, S. (2003). Using the Lunar Phases Concept Inventory to investigate college students’ pre-instructional mental models of lunar phases. AIP Conference Proceedings, 720 , 73-76.

Lopez, E., Kim, J., Nandagopal, K., Cardin, N., Shavelson, R.J., and Penn, J.H. (2011). Validating the use of concept-mapping as a diagnostic assessment tool in organic chemistry: Implications for teaching. Chemistry Education Research and Practice , 12 , 133-141.

Marsden, P., and Wright, J. (Eds.). (2010). Handbook of survey research (2nd ed.). Bingley, UK: Emerald.

McConnell, D.A., Steer, D.N., Owens, K., Borowski, W., Dick, J., Foos, A., Knott, J.R., Malone, M., McGrew, H., Van Horn, S., Greer, L., and Heaney, P.J. (2006). Using ConcepTests to assess and improve student conceptual understanding in introductory geoscience courses. Journal of Geoscience Education , 54 (1), 61-68.

McDermott, L.C., and Redish, E.F. (1999). Resource letter: PER-1: Physics education research. American Journal of Physics , 67 (9), 755-767.

McDermott, L.C., and Shaffer, P.S. (1992). Research as a guide for curriculum development: An example from introductory electricity. Part I: Investigation of student understanding. American Journal of Physics , 60 , 994-1003.

McNeal, K.S., Miller, H.R., and Herbert, B.E. (2008). Developing nonscience majors’ conceptual models of complex earth systems in a physical geology course. Journal of Geoscience Education , 56 , 201-211.

Minstrell, J. (1989). Teaching science for understanding. In L. Resnick and L. Klopfer (Eds.), 1989 Association for Supervision and Curriculum Development yearbook: Toward the thinking curriculum: Current cognitive research . Alexandria, VA: Association for Supervision and Curriculum Development.

Montfort, D., Brown, S., and Pollock, D. (2009). An investigation of students’ conceptual understanding in related sophomore to graduate-level engineering and mechanics courses. Journal of Engineering Education , 98 (2), 111-129.

Morabito, N.P., Catley, K.M., and Novick, L.R. (2010). Reasoning about evolutionary history: Postsecondary students’ knowledge of most recent common ancestry and homoplasy. Journal of Biological Education , 44 (4), 166-174.

Mulford, D.R., and Robinson, W.R. (2002). An inventory for alternate conceptions among first-semester general chemistry students. Journal of Chemical Education , 79 (6), 739-744.

Muryanto, S. (2006). Concept mapping: An interesting and useful tool for chemical engineering laboratories. International Journal of Engineering Education , 22 (5), 979-985.

Nakhleh, M.B. (1992). Why some students don’t learn chemistry: Chemical misconceptions. Journal of Chemical Education , 69 (3), 191-196.

National Research Council. (1999). How people learn: Bridging research and practice. M.S. Donovan, J.D. Bransford, and J.W. Pellegrino (Eds.). Committee on Learning Research and Educational Practice, Commission on Behavioral and Social Sciences and Education. Washington, DC: National Academy Press.

National Research Council. (2007). Taking science to school: Learning and teaching science in grades K-8. Committee on Science Learning, Kindergarten Through Eighth Grade, R.A. Duschl, H.A. Schweingruber, and A.W. Shouse (Eds.). Board on Science Education, Center for Education. Division of Behavioral and Social Sciences and Education Washington, DC: The National Academies Press.

Nicoll, G. (2003). A qualitative investigation of undergraduate chemistry students’ macroscopic interpretations of the submicroscopic structures of molecules. Journal of Chemical Education , 80 , 205-213.

Novak, J.D., and Gowin, D.B. (1984). Learning how to learn . New York: Cambridge University Press.

O’Brien, R.J., Lau, L.F., and Huganir, R.L. (1998) Molecular mechanisms of glutamate receptor clustering at excitatory synapses. Current Opinion Neurobiology, 8, 364-369.

Oreskes, N., Conway, E.M., and Shindell, M. (2008). From Chicken Little to Dr. Pangloss: William Nierenberg, global warming, and the social deconstruction of scientific knowledge. Historical Studies in the Natural Sciences , 38 (1), 109-152.

Orgill, M., and Sutherland, A. (2008). Undergraduate chemistry students’ perceptions of and misconceptions about buffers and buffer problems. Chemistry Education Research and Practice , 9 (2), 131-143.

Pelaez, N.J., Boyd, D.D., Rojas, J.B., and Hoover, M.A. (2005). Prevalence of blood circulation misconceptions among prospective elementary teachers. Advances in Physiology Education , 29 (3), 172-181.

Pintrich, P.R. (1999). Motivational beliefs as resources for and constraints on conceptual change. In W. Schnotz, S. Vosniadou, and M. Carretero (Eds.), New perspectives on conceptual change . Oxford, UK: Pergamon Press.

Plummer, J.D., and Krajcik, J. (2010). Building a learning progression for celestial motion: Elementary levels from an earth-based perspective. Journal of Research in Science Teaching , 47 (7), 768-787.

Posner, G.J., Strike, K.A., Hewson, P.W., and Gertzog, W.A. (1982). Accommodation of a scientific conception: Toward a theory of conceptual change. Science Education , 66 (2), 211-227.

Prince, M.J., Vigeant, M.A.S., and Nottis, K. (2009). A preliminary study on the effectiveness of inquiry-based activities for addressing misconceptions of undergraduate engineering students. Education for Chemical Engineers , 4 (2), 29-41.

Reed-Rhoads, T., and Imbrie, P.K. (2008). Concept inventories in engineering education . Paper presented at the National Research Council’s Workshop on Linking Evidence to Promising Practices in STEM Undergraduate Education, Washington, DC. Available: http://www7.nationalacademies.org/bose/Reed_Rhoads_CommissionedPaper.pdf .

Rushton, G.T., Hardy, R.C., Gwaltney, K.P., and Lewis, S.E. (2008). Alternative conceptions of organic chemistry topics among fourth year chemistry students. Chemistry Education Research and Practice , 9 (2), 122-130.

Schneps, M.H., and Sadler, P.M. (1987). A private universe . Available: http://www.learner.org/resources/series28.html .

Schwarz, C.V., Reiser, B.J., Davis, E.A., Kenyon, L., Achér, A., Fortus, D., Shwartz, Y., Hug, B., and Krajcik, J. (2009). Developing a learning progression for scientific modeling: Making scientific modeling accessible and meaningful for learners. Journal of Research in Science Teaching , 46 (6), 632-368.

Segalas, J., Ferrer-Balas, D., and Mulder, K. (2008). Conceptual maps: Measuring learning processes of engineering students concerning sustainable development. European Journal of Engineering Education , 33 (3), 297-306.

Sinatra, G.M., Southerland, S.A., McConaughy, F., and Demastes, J.W. (2003). Intentions and beliefs in students’ understanding and acceptance of biological evolution. Journal of Research in Science Teaching , 40 (5), 510-528.

Smith, M.K., Wood, W.B., and Knight, J.K. (2008). The genetics concept assessment: A new concept inventory for gauging student understanding of genetics. CBE-Life Sciences Education , 7 (4), 422-430.

Sokoloff, D.R., and Thornton, R.K. (1997). Using interactive lecture demonstrations to create an active learning environment. The Physics Teacher , 35 (6), 340-347.

Sözbilir, M. (2002). Turkish chemistry undergraduate students’ misunderstandings of Gibbs free energy. University Chemistry Education , 6 (2), 73-83.

Sözbilir, M. (2004). What makes physical chemistry difficult? Perceptions of Turkish chemistry undergraduates and lecturers. Journal of Chemical Education , 81 (4), 573-578.

Sözbilir, M., and Bennett, J.M. (2006). Turkish prospective chemistry teachers’ misunderstandings of enthalpy and spontaneity. Chemical Educator , 11 (5), 355-363.

Stevens, R., O’Connor, K., Garrison, L., Jocuns, A., and Amos, D.M. (2008). Becoming an engineer: Toward a three-dimensional view of engineering learning. Journal of Engineering Education , 97 (3), 355-368.

Taber, K.S. (2001) The mismatch between assumed prior knowledge and the learner’s conceptions: a typology of learning impediments, Educational Studies , 27 (2), 159-171.

Talanquer, V. (2002). Minimizing misconceptions: Tools for identifying patterns of reasoning. The Science Teacher , 69 , 46-49.

Talanquer, V. (2006). Common-sense chemistry: A model for understanding students’ alternative conceptions. Journal of Chemical Education , 83 , 811.

Tanner, K., and Allen, D. (2005). Approaches to biology teaching and learning: Understanding the wrong answers—teaching toward conceptual change. Cell Biology Education , 4 (2), 112-117.

Taraban, R., Anderson, E.E., DeFinis, A., Brown, A.G., Weigold, A., and Sharma, M.P. (2007). First steps in understanding engineering students’ growth of conceptual and procedural knowledge in an interactive learning context. Journal of Engineering Education , 96 (1), 57-68.

Teed, R., and Slattery, W. (2011). Changes in geologic time understanding in a class for pre-service teachers, Journal of Geoscience Education , 59, 151-162.

Treagust, D. (1988). Teaching science for conceptual change: Theory and practice. In S. Vosniadou (Ed.), International handbook of research on conceptual change (pp. 629-646). New York: Routledge.

Trend, R. (2000). Conceptions of geological time among primary teacher trainees, with reference to their engagement with geoscience, history, and science. International Journal of Science Education , 22 (5), 539-555.

Turns, J., Atman, C., Adams, R., and Barker, T. (2005). Research on engineering student knowing: Trends and opportunities. Journal of Engineering Education , 94 (1), 27-41.

Vosniadou, S. (2008a). Conceptual change research: An introduction. In S. Vosniadou (Ed.), International handbook of research on conceptual change (pp. xii-xxviii). New York: Routledge.

Vosniadou, S. (Ed.). (2008b). International handbook of research on conceptual change . New York: Routledge.

Vosniadou, S., Vamvakoussi, X., and Skopeliti, I. (2008). The framework theory approach to the problem of conceptual change. In S. Vosniadou (Ed.), International handbook of research on conceptual change (pp. 3-34). New York: Routledge.

Wandersee, J.H., Mintzes, J.J., and Novak, J.D. (1994). Research on alternative conceptions in science. In D. Gabel (Ed.), Handbook of research in science teaching and learning (pp. 177-210). New York: Simon & Schuster Macmillan.

Yezierski, E.J., and Birk, J.P. (2006). Misconceptions about the particulate nature of matter. Using animations to close the gender gap. Journal of Chemical Education , 83 (6), 954-960.

Zeilik, M. (2002). Birth of the Astronomy Diagnostic Test: Prototest evolution. Astronomy Education Review , 1 (2), 46.

Zeilik, M., and Bisard, W. (2000). Conceptual change in introductory-level astronomy courses. Journal of College Science Teaching , 29 (4), 229-232.

Zeilik, M., Schau, C., Mattern, N., Hall, S., Teague, K.W., and Bisard, W. (1997). Conceptual astronomy: A novel model for teaching postsecondary science courses. American Journal of Physics , 65 (10), 987-996.

Zeilik, M., Mattern, N., and Schau, C. (1999). Conceptual astronomy II. Replicating conceptual gains, probing attitude changes across three semesters. American Journal of Physics , 67 (10), 923-927.

Zimrot, R., and Ashkenazi, G. (2007). Interactive lecture demonstrations: A tool for exploring and enhancing conceptual change. Chemistry Education Research and Practice , 8 , 197-211.

ABET. (1995). Vision for change: A summary report of the ABET/NSF/industry workshops . Baltimore, MD: ABET.

Abraham, M., Varghese, V., and Tang, H. (2010). Using molecular representations to aid student understanding of stereochemical concepts. Journal of Chemical Education , 87 (12), 1425-1429.

Ainsworth, S., Prain, V., and Tytler, R. (2011). Drawing to learn science. Science , 333 , 1096-1097.

Aldahmash, A.H., and Abraham, M.R. (2009). Kinetic versus static visuals for facilitating college students’ understanding of organic reaction mechanisms in chemistry. Journal of Chemical Education , 86 (12), 1442-1446.

Allard, F., and Starkes, J. (1991). Motor-skill experts in experts in sports, dance and other domains. In K.A. Ericsson and H.G. Smith (Eds.), Toward a general theory of expertise (pp. 126-152). Cambridge, UK: Cambridge University Press.

Anderson, W.L., Mitchell, S.M., and Osgood, M.P. (2008). Gauging the gaps in student problem-solving skills: Assessment of individual and group use of problem-solving strategies using online discussions. CBE-Life Sciences Education, 7, 254-262.

Anderson, W.L., Sensibaugh, C.A., Osgood, M.P., and Mitchell, S.M. (2011). What really matters: Assessing individual problem-solving performance in the context of biological sciences. International Journal for the Scholarship of Teaching and Learning, 5 (1), 1-20.

Anzai, Y. (1991). Learning and use of representations for physics expertise . In K. Ericson and J. Smith (Eds.), Toward a general theory of expertise: Prospects and limits (pp. 64-90). Cambridge, UK: Cambridge University Press.

Atman, C., Kilgore, D., and McKenna, A. (2008). Characterizing design learning: A mixed methods study of engineering designers’ use of language. Journal of Engineering Education, 97 (3), 309-326.

Ault, C.R., Jr. (1994). Research on problem-solving: Earth science. In D. Gabel (Ed.), Handbook of research on science teaching and learning . New York: MacMillan.

Baddeley, A. (2007). Working memory , thought , and action. Oxford, UK: Oxford University Press.

Bagno, E., and Eylon, B. (1997). From problem solving to a knowledge structure: An example from the domain of electromagnetism. The American Journal of Physics, 65 (8), 726-736.

Baillie, C., Goodhew, P., and Skryabina, E. (2006). Threshold concepts in engineering education: Exploring potential blocks in student understanding. International Journal of Engineering Education , 22 (5), 955-962.

Barrows, H.S. (1986). A taxonomy of problem-based learning methods. Medical Education , 20 , 481-486.

Bassok, M., and Olseth, K.L. (1995). Object-based representations: Transfer between cases of continuous and discrete models of change. Journal of Experimental Psychology: Learning , Memory , and Cognition , 21 (6), 1522-1538.

Bassok, M., Chase, V.M., and Martin, S.A. (1998). Adding apples and oranges: Alignment of semantic and formal knowledge. Cognitive Psychology , 35 (2), 99-134.

Beichner, R.J. (1994). Testing student interpretation of kinematics graphs. American Journal of Physics , 62 (8), 750-762.

Bodner, G.M. (2004). Twenty years of learning: How to do research in chemical education. Journal of Chemical Education , 81 (5), 618-628.

Bodner, G.M., and Herron, J.D. (2002). Problem solving in chemistry. In J.K. Gilbert (Ed.), Chemical education: Research-based practice . Dordecht, The Netherlands: Kluwer Academic.

Bodner, G.M., and McMillan, T.L.B. (1986). Cognitive restructuring as an early stage in problem-solving. Journal of Research in Science Teaching , 23 (8), 727-737.

Bransford, J., Vye, N., and Bateman, H. (2002). Creating high-quality learning environments: Guidelines from research on how people learn. In National Research Council, The knowledge economy and postsecondary education: Report of a workshop (pp. 159-197). Committee on the Impact of the Changing Economy of the Education System. P.A. Graham and N.G. Stacey (Eds.). Center for Education, Division of Behavioral and Social Sciences and Education. Washington, DC: National Academy Press.

Brookes, D.T., and Etkina, E. (2007). Using conceptual metaphor and functional grammar to explore how language used in physics affects student learning. Physical Review Special Topics—Physics Education Research , 3 (1), 010105-1-010105-16.

Bunce, D.M., and Gabel, D. (2002). Differential effects on the achievement of males and females of teaching the particulate nature of chemistry. Journal of Research in Science Teaching , 39 (10), 911-927.

Bunce, D.M., and Heikkinen, H. (1986). The effects of an explicit problem-solving approach on mathematical chemistry achievement. Journal of Research in Science Teaching , 23 (1), 11-20.

Camacho, M., and Good, R. (1989). Problem solving and chemical equilibrium: Successful versus unsuccessful performance. Journal of Research in Science Teaching , 26 (3), 251-272.

Carlson, C.A. (1999). Field research as a pedagogical tool for learning hydrogeochemistry and scientific-writing skills. Journal of Geoscience Education , 47 (2), 150-157.

Catley, K.M., and Novick, L.R. (2008). Seeing the wood for the trees: An analysis of evolutionary diagrams in biology textbooks. Bioscience , 58 (10), 976-987.

Chandrasegaran, A.L., Treagust, D.F., Waldrip, B.G., and Chandrasegaran, A. (2009). Students’ dilemmas in reaction stoichiometry problem solving: Deducing the limiting reagent in chemical reactions. Chemistry Education Research and Practice , 10 (1), 14-23.

Chase, W.G., and Simon, H.A. (1973). The mind’s eye in chess. In W.G. Chase (Ed.), Visual information processing (pp. 215-281). New York: Academic Press.

Cheville, A. (2010). Transformative experiences: Scaffolding design learning through the Vygotsky cycle. International Journal of Engineering Education, 26 (4), 760-770.

Chi, M.T.H. (2006). Two approaches to the study of experts’ characteristics. In K.A. Ericsson, N. Charness, R.R. Hoffman, and P.J. Feltovich (Eds.), The Cambridge handbook of expertise and expert performance (pp. 21-30). New York: Cambridge University Press.

Chi, M.T.H., Feltovich, P.J., and Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science , 5 (2), 121-152.

Chi, M.T.H., Glaser, R., and Rees, E. (1982). Expertise in problem solving. In R. Sternberg (Ed.), Advances in the psychology of human intelligence (vol. 1, pp. 7-76). Hillsdale, NJ: Lawrence Erlbaum Associates.

Chi, M.T.H., Bassok, M., Lewis, M.W., Reimann, P., and Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science , 13 (2), 145-182.

Clement, J. (2009). Creative model construction in scientists and students. The role of imagery , analogy and mental simulations. Dordrecht, The Netherlands: Springer.

Cohen, E., Mason, A., Singh, C., and Yerushalmi, E. (2008). Identifying differences in diagnostic skills between physics students: Students’ self-diagnostic performance given alternative scaffolding. AIP Conference Proceedings, 1064, 99-102.

Collins, A., Brown, J.S., and Newman, S.E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In L.B. Resnick (Ed.), Knowing , learning , and instruction: Essays in honor of Robert Glaser (pp. 453-494). Hillsdale, NJ: Lawrence Erlbaum Associates.

Connor, C. (2009). Field glaciology and earth systems science: The Juneau Icefield Research Program (JIRP), 1946-2008. Geological Society of America Special Papers, 461 , 173-184.

Cooper, M.M., Cox, C.T., Jr., Nammouz, M., Case, E., and Stevens, R.J. (2008). An assessment of the effect of collaborative groups on students’ problem-solving strategies and abilities. Journal of Chemical Education , 85 (6), 866-872.

Cooper, M., Grove, N., Underwood, S., and Klymkowsky, M. (2010). Lost in Lewis structures: An investigation of student difficulties in developing representational competence. Journal of Chemical Education , 87 (8), 869-874.

Cordray, D.S., Harris, T.R., and Klein, S. (2009). A research synthesis of the effectiveness, replicability, and generality of the VaNTH: Challenge-based instructional modules in bioengineering. Journal of Engineering Education , 98 (4), 335-348.

Coughlin, L.D., and Patel, V.L. (1987). Processing of critical information by physicians and medical students. Journal of Medical Education , 62 (10), 818-828.

Cummings, K., Marx, J., Thornton, R., and Kuhl, D. (1999). Evaluating innovation in studio physics. American Journal of Physics , 67 (7, Suppl. 1), S38-S44.

de Jong, T., and Ferguson-Hessler, M.G.M. (1986). Cognitive structures of good and poor novice problem solvers in physics. Journal of Educational Psychology, 78 , 279-288.

de Wet, A., Manduca, C., Wobus, R.A., and Bettision-Varga, L. (2009). Twenty-two years of undergraduate research in the geosciences—the Keck experience. Geological Society of America Special Papers, 461 , 163-172.

Docktor, J.L., and Heller, K. (2009). Assessment of student problem solving processes. AIP Conference Proceedings, 1179, 133-136.

Duch, B.J. (1997). Problem-based learning in physics: Making connections with the real world. AIP Conference Proceedings, 399, 557-559.

Dufour-Janvier, B., Bednarz, N., and Belanger, M. (1987). Pedagogical considerations concerning the problem of representation. In C. Janvier (Ed.), Problems of representation in the teaching and learning of mathematics (pp. 109-122). Hillsdale, NJ: Lawrence Erlbaum Associates.

Dufresne, R.J., Gerace, W.J., Hardiman, P.T., and Mestre, J.P. (1992). Constraining novices to perform expert-like problem analyses: Effects on schema acquisition. Journal of the Learning Sciences, 2 , 307-331.

Dutrow, B.L. (2007). Visual communication: Do you see what I see? Elements , 3 (2), 119-126.

Dutson, A., Todd, R., Magelby, S., and Sorenson, C. (1997). A review of literature on teaching engineering design through project-oriented capstone courses. Journal of Engineering Education , 86 (1), 17-28.

Dym, C.L., Agogino, A.M., Eris, O., Frey, D.D., and Leifer, L.J. (2005). Engineering design thinking, teaching, and learning. Journal of Engineering Education , 94 (1), 103-119.

Eastman, C. (2001). New directions in design cognition: Studies of representation and recall. In C. Eastman, W. Newstetter, and M. McCracken (Eds.), Design knowing and learning: Cognition in design education (pp. 147-198). Oxford, UK: Elsevier Science.

Egan, D.E., and Schwartz, B.J. (1979). Chunking in recall of symbolic drawings. Memory and Cognition , 7 (2), 149-158.

Ericsson, K.A., and Simon, H.A. (1980). Verbal reports as data. Psychological Review , 87 , 215-251.

Ericsson, K.A., and Simon, H.A. (1993). Protocol analysis: Verbal reports as data (rev. ed.). Cambridge, MA: MIT Press.

Eylon, B., and Reif, F. (1984). Effects of knowledge organization on task performance. Cognition and Instruction, 1, 5-44.

Fay, M.E., Grove, N.P., Towns, M.H., and Bretz, S.L. (2007). A rubric to characterize inquiry in the undergraduate chemistry laboratory. Chemistry Education Research and Practice , 8 (2), 212-219.

Ferrini-Mundy, J., and Graham, K. (1994). Research in calculus learning: Understanding of limits, derivatives, and integrals. In J. Kaput and E. Dubinsky (Eds.), Research issues in undergraduate mathematics learning: Preliminary analyses and results (pp. 31-35). Washington, DC: Mathematical Association of America.

Finegold, M., and Mass, R. (1985). Differences in the process of solving physics problems between good problem solvers and poor problem solvers. Research in Science and Technology Education, 3, 59-67.

Fuhrman, O., and Boroditsky, L. (2010). Cross-cultural differences in mental representations of time: Evidence from an implicit non-linguistic task. Cognitive Science, 34, 1430-1451.

Fuller, I., Edmondson, S., France, D., Higgitt, D., and Ratinen, I. (2006). International perspectives on the effectiveness of geography fieldwork for learning. Journal of Geography in Higher Education , 30 (1), 89-101.

Gabel, D., and Bunce, D. (1994). Research on problem solving: Chemistry. In D. Gabel (Ed.), Handbook of research on science teaching and learning (pp. 301-326). New York: Macmillan.

Gerson, H.B.P., Sorby, S.A., Wysocki, A., and Baartmans, B.J. (2001). The development and assessment of multimedia software for improving 3-D visualization skills. Computer Applications in Engineering Education, 9, 105-113.

Gertner, A.S., and VanLehn, K. (2000). Andes: A coached problem-solving environment for physics. In G. Gautheier, C. Frasson, and K. VanLehn (Eds.), Intelligent tutoring systems: 5th international conference , ITS 2000 (pp. 133-142). New York: Springer.

Gilbert, J.K., and Treagust, D.F. (Eds.). (2009). Multiple representations in chemical education . New York: Springer.

Glasgow, J., Narayanan, N.H., and Chandrasekaran, B. (1995). Diagrammatic reasoning: Cognitive and computational perspective s (pp. 303-338). Menlo Park, CA: AAAI Press.

Gobet, F., and Simon, H.A. (1996). The roles of recognition processes and look-ahead search in time-constrained expert problem solving: Evidence from grand-master-level chess. Psychological Science , 7 (1), 52-55.

Goldberg, F.M., and Anderson, J.H. (1989). Student difficulties with graphical representations of negative values of velocity. Physics Teacher , 27 (4), 254-260.

Gonzales, D., and Semken, S. (2009). Using field-based research studies to engage undergraduate students in igneous and metamorphic petrology: A comparative study of pedagogical approaches and outcomes. Geological Society of America Special Papers, 461, 205-221.

Goodwin, C. (1994). Professional vision. American Anthropologist , 96 (3), 606-633.

Hardiman, P.T., Dufresne, R., and Mestre, J.P. (1989). The relation between problem categorization and problem solving among experts and novices. Memory and Cognition , 17 (5), 627-638.

Harris, M.A., Peck, R.F., Colton, S., Morris, J., Chaibub Neto, E., and Kallio, J. (2009). A combination of hand-held models and computer imaging programs helps students answer oral questions about molecular structure and function: A controlled investigation of student learning. CBE-Life Sciences Education, 8, 29-43.

Hayes, J.R., and Simon, H.A. (1977). Psychological differences among problem isomorphs. In N.J. Castellan, D.B. Pisoni, and G.R. Potts (Eds.), Cognitive theory (vol. 2, pp. 21-44). Hillsdale, NJ: Lawrence Erlbaum Associates.

Hegarty, M. (1992). Mental animation: Inferring motion from static diagrams of mechanical systems. Journal of Experimental Psychology: Learning , Memory and Cognition , 18 (5), 1084-1102.

Hegarty, M. (2011). The role of spatial thinking in undergraduate science education . Paper presented at the Third Committee Meeting on Status, Contributions, and Future Directions of Discipline-Based Education Research. Available: http://www7.nationalacademies.org/bose/DBER_Hegarty_December_Paper.pdf .

Hegarty, M., and Sims, V.K. (1994). Individual differences in mental animation during mechanical reasoning. Memory and Cognition , 22 (4), 411-430.

Hegarty, M., Carpenter, P.A., and Just, M.A. (1991). Diagrams in the comprehension of scientific texts. In R. Barr, M.L. Kamil, P. Mosenthal, and P.D. Pearson (Eds.), Handbook of reading research. New York: Longman.

Hegarty, M., Kriz, S., and Cate, C. (2003). The roles of mental animations and external animations in understanding mechanical systems. Cognition and Instruction , 21 (4), 209-249.

Hegarty, M., Crookes, R., Dara-Abrams, D., and Shipley, T.F. (2010). Do all science disciplines rely on spatial abilities? Preliminary evidence from self-report questionnaires. In C. Hölscher, T.E. Shipley, M.O. Belardanelli, J.A. Bateman, and N.S. Newcombe (Eds.), Proceedings of spatial cognition 2010 (pp. 85-94). Berlin, Germany: Springer.

Heller, J., and Reif, F. (1984). Prescribing effective human problem-solving processes: Problem description in physics. Cognition and Instruction, 1 (2), 177-216.

Heller, K., and Heller, P. (2000). Competent problem solver—calculus version . New York: McGraw-Hill.

Heller, P., and Hollabaugh, M. (1992). Teaching problem-solving through cooperative grouping. Part 2: Designing problems and structuring groups. American Journal of Physics , 60 (7), 637-644.

Heller, P., Keith, R., and Anderson, S. (1992). Teaching problem solving through cooperative grouping. Part 1: Group versus individual problem solving. American Journal of Physics , 60 (7), 627-636.

Herron, J.D., and Greenbowe, T.J. (1986). What can we do about Sue: A case study of competence. Journal of Chemical Education, 63 , 526-531.

Hoellwarth, C., Moelter, M.J., and Knight, R.D. (2005). A direct comparison of conceptual learning and problem-solving ability in traditional and studio style classrooms. American Journal of Physics , 73 (5), 459-462.

Hsu, L., and Heller, K. (2009). Computer problem-solving coaches . Proceedings of the National Association for Research in Science Teaching 82nd International Conference, Garden Grove, CA.

Hsu, L., Brewe, E., Foster, T.M., and Harper, K.A. (2004). Resource letter RPS-1: Research in problem solving. American Journal of Physics , 72 (9), 1147-1156.

Hundhausen, C., Agarwal, P., Zollars, R., and Carter, A. (2011). The design and experimental evaluation of a scaffolded software environment to improve engineering students’ disciplinary problem-solving skills. Journal of Engineering Education, 100 (3), 574-603.

Hurley, S.M., and Novick, L.R. (2010). Solving problems using matrix, network, and hierarchy diagrams: The consequences of violating construction conventions. Quarterly Journal of Experimental Psychology , 63 (2), 275-290.

Hynd, C., McWhorter, J.Y., Phares, V.L., and Suttles, C.W. (1994). The role of instructional variables in conceptual change in high school physics topics. Journal of Research in Science Teaching , 31 (9), 933-946.

Isaak, M.I., and Just, M.A. (1995). Constraints on the processing of rolling motion: The curtate cycloid illusion. Journal of Experimental Psychology: Human Perception and Performance , 21 (6), 1391-1408.

Ishikawa, T., Barnston, A.G., Kastens, K.A., Louchouarn, P., and Ropelewski, C.F. (2005). Climate forecast maps as a communication and decision-support tool: An empirical test with prospective policy makers. Cartography and Geographic Information Science , 32 (1), 3-16.

Ishikawa, T., Barnston, A.G., Kastens, K.A., and Louchouarn, P. (2011). Understanding, evaluation, and use of climate forecast data by environmental policy students. Geological Society of America Special Papers , 474 , 153-170.

Johnson, D.W., Johnson, R.T., and Smith, K.A. (1991). Cooperative learning: Increasing college faculty instructional productivity. ASHE-ERIC Higher Education Report No. 4. Washington, DC: The George Washington University School of Education and Human Development.

Johnstone, A.H. (1991). Why is science difficult to learn? Things are seldom what they seem. Journal of Computer Assisted Learning, 7, 75-83.

Johnstone, A.H., and El-Banna, H. (1986). Capacities, demands and processes: A predictive model for science education. Education in Chemistry , 23 , 80-84.

Kali, Y., and Orion, N. (1996). Spatial abilities of high-school students in the perception of geologic structures. Journal of Research in Science Teaching , 33 (4), 369-391.

Kastens, K. (2009, October). Synthesis of research on thinking and learning in the geosciences: Developing representational competence . Paper presented at the annual meeting of the Geological Society of America. Abstract available: https://gsa.confex.com/gsa/2009AM/finalprogram/abstract_165183.htm .

Kastens, K. (2010). Object and spatial visualization in geosciences. Journal of Geoscience Education , 58 (2), 52-57.

Kastens, K.A., and Manduca, C.A. (2012). Fostering knowledge integration in geoscience education. Geological Society of America Special Papers , 486 , 183-206.

Kellman, P.J. (2000). An update on Gestalt psychology. In B. Landau, J. Sabini, J. Jonides, and E. Newport (Eds.), Perception , cognition , and language: Essays in honor of Henry and Lila Gleitman. Cambridge, MA: MIT Press.

Kelly, R.M., and Jones, L.L. (2008). Investigating students’ ability to transfer ideas learned from molecular animations of the dissolution process. Journal of Chemical Education , 85 (2), 303-309.

Kindfield, A.C.H. (1993/1994). Biology diagrams: Tools to think with. Journal of the Learning Sciences, 3, 1-36.

Klein, G.A., Orasanu, J., Calderwood, R., and Zsambok, C.E. (Eds.). (1993). Decision making in action: Models and methods . Norwood, NJ: Ablex.

Kohl, P.B., and Finkelstein, N.D. (2005). Student representational competence and self-assessment when solving physics problems. Physical Review Special Topics—Physics Education Research , 1 (1), 010104-1–010104-11.

Kohl, P.B., Rosengrant, D., and Finkelstein, N.D. (2007). Strongly and weakly directed approaches to teaching multiple representation use in physics. Physical Review Special Topics—Physics Education Research , 3 (1), 010108-1–010108-10.

Kolb, D.A. (1984). Experiential learning . Englewood Cliffs, NJ: Prentice Hall.

Kotovsky, K., Hayes, J.R., and Simon, H.A. (1985). Why are some problems hard? Evidence from tower of Hanoi. Cognitive Psychology , 17 (2), 248-294.

Kowalski, P., and Taylor, A.K. (2009). The effect of refuting misconceptions in the introductory psychology class. Teaching of Psychology , 36 (3), 153-159.

Kozhevnikov, M., and Thornton, R. (2006). Real-time data display, spatial visualization ability, and learning force and motion concepts. Journal of Science Education and Technology , 15 (1), 111-132.

Kozhevnikov, M., Hegarty, M., and Mayer, R.E. (2002). Revising the visualizer-verbalizer dimension: Evidence for two types of visualizers. Cognition and Instruction , 20 (1), 47-77.

Kozma, R.B., and Russell, J. (1997). Multimedia and understanding: Expert and novice responses to different representations of chemical phenomena. Journal of Research in Science Teaching , 34 (9), 949-968.

Kriz, S., and Hegarty, M. (2007). Top-down and bottom-up influences on learning from animations. International Journal of Human Computer Studies , 65 (11), 911-930.

Lajoie, S.P. (2003). Transitions and trajectories for studies of expertise. Educational Researcher, 32 (8), 21-25.

Landy, D., and Goldstone, R.L. (2007). How abstract is symbolic thought? Journal of Experimental Psychology: Learning Memory and Cognition , 33 (4), 720-733.

Larkin, J.H. (1979). Processing information for effective problem solving. Engineering Education , 70 (3), 285-288.

Larkin, J.H. (1981a). Cognition of learning physics. American Journal of Physics, 49 (6), 534-541.

Larkin, J.H. (1981b). Enriching formal knowledge: A model for learning to solve textbook physics problems. In J.R. Anderson (Ed.), Cognitive skills and their acquisition (pp. 311-334). Hillsdale, NJ: Lawrence Erlbaum Associates.

Larkin, J.H. (1983). Role of problem representation in physics. In D. Gentner and A. Stevens (Eds.), Mental models (pp. 75-98). Hillsdale, NJ: Lawrence Erlbaum Associates.

Larkin, J.H., and Simon, H.A. (1987). Why a diagram is (sometimes) worth ten thousand words. Cognitive Science , 11 (1), 65-100.

Larkin, J.H., McDermott, J., Simon, D.P., and Simon, H.A. (1980). Models of competence in solving physics problems. Cognitive Science , 4 (4), 317-345.

Lave, J., and Wenger, E. (1991). Situated learning: Legitimate peripheral participation . Cambridge, MA: Cambridge University Press.

Lewis, S.E., and Lewis, J.E. (2005). Departing from lectures: An evaluation of a peer-led guided inquiry alternative. Journal of Chemical Education , 82 (1), 135-139.

Libarkin, J., and Brick, C. (2002). Research methodologies in science education: Visualization and the geosciences. Journal of Geoscience Education , 50 , 449-455.

Liben, L.S. (1997). Children’s understanding of spatial representations of place: Mapping the methodological landscape. In N. Foreman and R. Gillett (Eds.), A handbook of spatial research paradigms and methodologies , Vol. 1: Spatial cognition in the child and adult (pp. 41-82). East Sussex, UK: The Psychology Press (Taylor and Francis Group).

Liben, L.S., Kastens, K.A., and Christensen, A. (2011). Spatial foundations of science education: The illustrative case of instruction on introductory geological concepts. Cognition and Instruction , 29 (1), 1-43.

Linenberger, K.J., and Bretz, S.L. (2012). Generating cognitive dissonance in student interviews through multiple representations. Chemistry Education Research and Practice, 13 (3), 172-178.

Litzinger, T., Van Meter, P., Kapli, N., Zappe, S., and Toto, R. (2010). Translating education research into practice within an engineering education center: Two examples related to problem solving. International Journal of Engineering Education , 26 (4), 860-868.

Lynch, M. (1990). The externalized retina: Selection and mathematization in the visual documentation of objects in the life sciences. In M. Lynch and S. Woolgar (Eds.), Representation in scientific practice (pp. 153-186). Cambridge, MA: MIT Press.

Macdonald, R.H., Manduca, C.A., Mogk, D.W., and Tewksbury, B.J. (2005). Teaching methods in undergraduate geoscience courses: Results of the 2004 On the Cutting Edge survey of U.S. faculty. Journal of Geoscience Education , 53 (3), 237-252.

Manduca, C., and Kastens, K.A. (2012). Geoscience and geoscientists: Uniquely equipped to study the Earth. Geological Society of America Special Papers, 486 , 1-12.

Markman, A.B. (1999). Knowledge representation . Mahwah, NJ: Lawrence Erlbaum Associates.

Marshall, S.P. (1995). Schemas in problem solving . New York: Cambridge University Press.

Martin, S.A., and Bassok, M. (2005). Effects of semantic cues on mathematical modeling: Evidence from word-problem solving and equation construction tasks. Memory and Cognition , 33 (3), 471-478.

Martinez, M.E. (2010). Learning and cognition: The design of the mind. Upper Saddle River, NJ: Merrill.

May, C.L., Eaton, L.S., and Whitmeyer, S.J. (2009). Integrating student-led research in fluvial geomorphology into traditional field courses: A case study from James Madison University’s field course in Ireland. Geological Society of America Special Papers , 461, 195-204.

McClary, L., and Talanquer, V. (2010). College chemistry students’ mental models of acids and acid strength. Journal of Research in Science Teaching , 48 (1), 396-413.

McCraken, W.M., and Newstetter, W. (2001). Text to diagram to symbol: Representational transformations in problem-solving . Proceedings of the American Society of Engineering Education/IEEE Frontiers in Education Conference, Reno, NV.

McDermott, L.C., Rosenquist, M.L., and van Zee, E.H. (1987). Student difficulties in connecting graphs and physics: Examples from kinematics. American Journal of Physics 55 (6), 503-513.

McKeithen, K.B., Reitman, J.S., Rueter, H.H., and Hirtle, S.C. (1981). Knowledge organization and skill differences in computer programmers. Cognitive Psychology , 13 (3), 307-325.

McKim, R.H. (1980). Thinking visually: A strategy manual for problem solving . Belmont, CA: Lifetime Learning.

McLean, P., Johnson, C., Rogers, R., Daniels, L., Reber, J., Slator, B.M., Terpstra, J., and White, A. (2005). Molecular and cellular biology animations: Development and impact on student learning. Cell Biology Education, 4, 169-179.

Meltzer, D.E. (2005). Relation between students’ problem-solving performance and representational format. American Journal of Physics , 73 (5), 463-478.

Mestre, J.P. (2002). Probing adults’ conceptual understanding and transfer of learning via problem posing. Journal of Applied Developmental Psychology , 23 (1), 9-50.

Mestre, J.P., and Ross, B.H. (Eds.). (2011). The psychology of learning and motivation (vol. 55). San Diego, CA: Elsevier.

Meyer, J.H.F., and Land, R. (2005). Threshold concepts and troublesome knowledge (2): Epistemological considerations and a conceptual framework for teaching and learning. Higher Education , 49 (3), 373-388.

Murphy, R., Ormand, C., Goodwin, L., Shipley, T., Manduca, C., and Tikoff, B. (2011, October). Improving students’ visuo-penetrative thinking skills through brief , weekly practice . Paper presented at the Geological Society of America Annual Meeting, Minneapolis, MN.

Nachshon, I. (1985). Directional preferences in perception of visual stimuli. International Journal of Behavioral Neuroscience, 25 , 161-174.

Nakhleh, M.B., and Mitchell, R.C. (1993). Concept learning versus problem solving: There is a difference. Journal of Chemical Education , 70 (3), 190-192.

Niaz, M. (1989). Dimensional analysis: A neo-piagetian evaluation of m-demand of chemistry problems. Research in Science & Technological Education, 7 (2), 153-170.

Novick, L.R. (1988). Analogical transfer, problem similarity, and expertise. Journal of Experimental Psychology: Learning , Memory , and Cognition , 14 (3), 510-520.

Novick, L.R. (2001). Spatial diagrams: Key instruments in the toolbox for thought. In D.L. Medin (Ed.), The psychology of learning and motivation (vol. 40, pp. 279-325). San Diego, CA: Academic Press.

Novick, L.R., and Bassok, M. (2005). Problem solving. In K.J. Holyoak and R.G. Morrison (Eds.), The Cambridge handbook of thinking and reasoning (pp. 321-349). New York: Cambridge University Press.

Novick, L.R., and Catley, K.M. (2007). Understanding phylogenies in biology: The influence of a gestalt perceptual principle. Journal of Experimental Psychology: Applied , 13 (4), 197-223.

Novick, L.R., and Sherman, S.J. (2008). The effects of superficial and structural information on online problem solving for good versus poor anagram solvers. Quarterly Journal of Experimental Psychology , 61 (7), 1098-1120.

Novick, L.R., Catley, K.M., and Funk, D.J. (2010). Characters are key: The effect of synapomorphies on cladogram comprehension. Evolution: Education and Outreach , 3 , 539-547.

Novick, L.R., Catley, K.M., and Funk, D.J. (in press). Inference is bliss: Using evolutionary relationship to guide categorical inferences. Cognitive Science .

Novick, L.R., Catley, K.M., and Schreiber, E.G. (2010). Understanding evolutionary history: An introduction to tree thinking (version 3). Unpublished instructional booklet, Department of Psychology and Human Development. Nashville, TN: Vanderbilt University.

Novick, L.R., Stull, A.T., and Catley, K.M. (in press). Reading Phylogenetic trees: Effects of tree orientation and text processing on comprehension. BioScience .

Nurrenbern, S.C., and Pickering, M. (1987). Concept-learning versus problem-solving: Is there a difference? Journal of Chemical Education , 64 (6), 508-510.

O’Day, D.H. (2007). The value of animations in biology teaching: A study of long-term memory retention. CBE-Life Sciences Education , 6 (3), 217-223.

Ogilvie, C.A. (2009). Changes in students’ problem-solving strategies in a course that includes context-rich, multifaceted problems. Physical Review Special Topics—Physics Education Research , 5 (2).

Orion, N., Ben-Chaim, D., and Kali, Y. (1997). Relationship between earth-science education and spatial visualization. Journal of Geoscience Education , 45 (2), 129-132.

Orton, A. (1983). Students’ understanding of integration. Educational Studies in Mathematics , 14 (1), 1-18.

Owen, E., and Sweller, J. (1985). What do students learn while solving mathematics problems? Journal of Educational Psychology , 77 , 272-284.

Ozdemir, G., Piburn, M., and Sharp, T. (2004). Exploring the use of visualization in a college mineralogy course . Paper presented at the National Association for Research in Science Teaching, Vancouver, BC.

Petcovic, H.L., and Libarkin, J.C. (2007). Research in science education: The expert-novice continuum. Journal of Geoscience Education , 55 (4), 333-339.

Piaget, J. (1978). Success and understanding . Cambridge, MA: Harvard University Press.

Piburn, M.D., Reynolds, S.J., McAuliffe, C., Leedy, D.E., and Johnson, J.K. (2005). The role of visualization in learning from computer-based images. International Journal of Science Education , 27 (5), 513-527.

Piburn, M.D., van der Hoeven Kraft, K., and Pacheco, H. (2011). A new century for geoscience education research . Paper presented at the Second Committee Meeting on the Status, Contributions, and Future Directions of Discipline-Based Education Research. Available: http://www7.nationalacademies.org/bose/DBER_Piburn_October_Paper.pdf .

Pólya, G. (1945). How to solve it , a new aspect of mathematical method . Princeton, NJ: Princeton University Press.

Posner, M.I. (1973). Cognition: An introduction . Glenview, IL: Scott, Foresman.

Potter, N. Jr., Niemitz, J.W., and Sak, P.B. (2009). Long-term field-based studies in geoscience teaching. Geological Society of America Special Papers, 461 , 185-194.

Previc, F.H. (1998). The neuropsychology of 3-D space. Psychological Bulletin , 124 , 123-164.

Pribyl, J.R., and Bodner, G.M. (1987). Spatial ability and its role in organic-chemistry: A study of four organic courses. Journal of Research in Science Teaching , 24 (3), 229-240.

Reif, F. (1995). Understanding basic mechanics . New York: John Wiley & Sons.

Reif, F., and Heller, J. (1982). Knowledge structure and problem solving in physics. Educational Psychologist, 17 (2), 102-127.

Reif, F., and Scott, L.A. (1999). Teaching scientific thinking skills: Students and computers coaching each other. American Journal of Physics , 67 (9), 819-831.

Reitman, W.R. (1965). Cognition and thought , an information-processing approach . New York: John Wiley & Sons.

Resnick, L.B. (1991). Shared cognition: Thinking as social practice. In L. Resnick, J. Levine, and S. Teasley (Eds.), Perspectives on socially shared cognition (pp. 127-149). Hyattsville, MD: American Psychological Association.

Riggs, E.M., Balliet, R., and Lieder, C. (2009). Using GPS tracking to understand problem solving during geologic field examinations. Geological Society of America Special Papers, 461 .

Riggs, E.M., Lieder, C.C., and Balliet, R. (2009). Geologic problem solving in the field: Analysis of field navigation and mapping by advanced undergraduates. Journal of Geoscience Education , 57 (1), 48-63.

Rosengrant, D., Van Heuvelen, A., and Etkina, E. (2005). Case study: Students’ use of multiple representations in problem solving. AIP Conference Proceedings, 818, 49-52.

Rosengrant, D., Etkina, E., and Van Heuvelen, A. (2007). An overview of recent research on multiple representations. AIP Conference Proceedings, 883, 149-152.

Sadler, T.D., Burgin, S., McKinney, L., and Ponjuan, L. (2010). Learning science through research apprenticeships: A critical review of the literature. Journal of Research in Science Teaching , 47 (3), 235-256.

Sandi-Urena, S., and Cooper, M.M. (2009). Design and validation of an inventory to assess metacognitive skillfulness in chemistry problem solving. Journal of Chemical Education , 86 , 240-245.

Sanger, M.J., and Badger, S.M. (2001). Using computer-based visualization strategies to improve students’ understanding of molecular polarity and miscibility. Journal of Chemical Education , 78 (10), 1412.

Sankar, C., Varma, V., and Raju, P. (2008). Use of case studies in engineering education: Assessment of changes in cognitive skills. Journal of Professional Issues in Engineering Education and Practice, 134 (3), 287-296.

Sawada, D., Piburn, M.D., Judson, E., Turley, J., Falconer, K., Benford, R., and Bloom, I. (2002). Measuring reform practices in science and mathematics classrooms: The reformed teaching observation protocol. School Science and Mathematics , 102 , 245-253.

Sawrey, B.A. (1990). Concept-learning versus problem-solving—revisited. Journal of Chemical Education , 67 (3), 253-254.

Schoenfeld, A.H., and Herrmann, D.J. (1982). Problem perception and knowledge structure in expert and novice mathematical problem solvers. Journal of Experimental Psychology-Learning Memory and Cognition , 8 (5), 484-494.

Schönborn, K.J., and Anderson, T.R. (2009). A model of factors determining students’ ability to interpret external representations in biochemistry. International Journal of Science Education, 31 (2), 193-232.

Schwartz, D.L., and Black, J.B. (1996). Shuttling between depictive models and abstract rules: Induction and fallback. Cognitive Science, 20 , 457-497.

Shea, D.L., Lubinski, D., and Benbow, C.P. (2001). Importance of assessing spatial ability in intellectually talented young adolescents: A 20-year longitudinal study. Journal of Educational Psychology , 93 (3), 604-614.

Simon, D.P., and Simon, H.A. (1978). Individual differences in solving physics problems. In R.S. Siegler (Ed.), Children’s thinking: What develops? (pp. 325-348). Hillsdale, NJ: Lawrence Erlbaum Associates.

Simon, H.A. (1978). Information-processing theory of human problem solving. In W.K. Estes (Ed.), Handbook of learning and cognitive processes (vol. 5, pp. 271-295). Hillsdale, NJ: Lawrence Erlbaum Associates.

Singh, K., Granville, M., and Dika, S. (2002). Mathematics and science achievement: Effects of motivation, interest, and academic engagement. The Journal of Educational Research, 95 (6), 323-332.

Smith, A.D., Mestre, J.P., and Ross, B.H. (2010). Eye-gaze patterns as students study worked-out examples in mechanics. Physical Review Special Topics—Physics Education Research , 6 (2).

Smith, M.U. (1992). Expertise and the organization of knowledge: Unexpected differences among genetic counselors, faculty, and students on problem categorization tasks. Journal of Research in Science Teaching , 29 , 179-205.

Smith, M.U., and Good, R. (1984). Problem solving and classical genetics: Successful versus unsuccessful performance. Journal of Research in Science Teaching, 21, 895-912.

Sorby, S.A. (2009). Educational research in developing 3-D spatial skills for engineering students. International Journal of Science Education , 31 (3), 459-480.

Stevens, R., and Palacio-Cayetano, J. (2003). Design and performance frameworks for constructing problem-solving simulation. Cell Biology Education, 2, 162-178.

Stieff, M. (2011). When is a molecule three-dimensional? A task-specific role for imagistic reasoning in advanced chemistry. Science Education, 95 (2), 310-336.

Stieff, M., Hegarty, M., and Dixon, B.L. (2010). Alternative strategies for spatial reasoning with diagrams. In A. Goel, M. Jamnik, and N.H. Narayanan (Eds.), Diagrammatic representation and inference ( Proceedings of Diagrams 2010 ). Berlin, Germany: Springer-Verlag.

Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12, 257-285.

Sweller, J., and Levine, M. (1982). Effects of goal specificity on means-ends analysis and learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 8, 463-474.

Sweller, J., Mawer, R., and Ward, M. (1983). Development of expertise in mathematical problem solving. Journal of Experimental Psychology: General, 112, 634-656.

Swenson, S., and Kastens, K.A. (2011). Student interpretation of a global elevation map: What it is, how it was made, and what it is useful for. Geological Society of America Special Papers, 474 , 189-211.

Taagepera, M., and Noori, S. (2000). Mapping students’ thinking patterns in learning organic chemistry by the use of knowledge space theory. Journal of Chemical Education, 77 (9), 1224-1229.

Taatgen, N.A., and Anderson, J.R. (2008). ACT-R. In R. Sun (Ed.), Constraints in cognitive architectures (pp. 170-185). Cambridge, MA: Cambridge University Press.

Taber, K. (2009). Learning at the symbolic level. In J.K. Gilbert and D.F. Treagust (Eds.), Multiple representations in chemical education (pp. 75-108). Dordrecht, The Netherlands: Springer.

Taylor, J.L., Smith, K.M., van Stolk, A.P., and Spiegelman, G.B. (2010). Using invention to change how students tackle problems. CBE-Life Sciences Education, 9, 504-512.

Thagard, P. (2005). Mind: Introduction to cognitive science (2nd ed.). Cambridge, MA: MIT Press.

Tien, L.T., Teichert, M.A., and Rickey, D. (2007). Effectiveness of a MORE laboratory module in prompting students to revise their molecular-level ideas about solutions. Journal of Chemical Education , 84 (1), 175-181.

Tingle, J.B., and Good, R.G. (1990). Effects of cooperative grouping on stoichiometric problem solving in high school chemistry. Journal of Research in Science Teaching , 27 (7), 671-683.

Titus, S., and Horsman, E. (2009). Characterizing and improving spatial visualization skills. Journal of Geoscience Education , 57 (4), 242-254.

Tversky, B., Zacks, J., Lee, P.U., and Heiser, J. (2000). Lines, blobs, crosses and arrows. In M. Anderson, P. Cheng, and V. Haarslev (Eds.), Theory and application of diagrams (pp. 221-230). Berlin, Heidelberg: Springer-Verlag.

Tversky, B., Morrison, J.B., and Betrancourt, M. (2002). Animation: Can it facilitate? International Journal of Human-Computer Studies , 57 , 247-262.

Van Heuvelen, A. (1991). Learning to think like a physicist: A review of research-based instructional strategies. American Journal of Physics , 59 (10), 891-897.

Van Heuvelen, A. (1995). Experiment problems for mechanics. The Physics Teacher , 33 , 176-180.

Van Heuvelen, A., and Etkina, E. (2006). The physics active learning guide . San Francisco, CA: Pearson, Addison Wesley.

Van Heuvelen, A., and Maloney, D. (1999). Playing physics jeopardy. American Journal of Physics , 67 , 252-256.

Van Heuvelen, A., and Zou, X. (2001). Multiple representations of work and energy processes. American Journal of Physics, 69 (2), 184-194.

Voss, J.F., Greene, T.R., Post, T., and Penner, B.C. (1983). Problem-solving skills in the social sciences. In G. Bower (Ed.), The psychology of learning and motivation (pp. 165-213). New York: Academic Press.

Wai, J., Lubinski, D., and Benbow, C.P. (2009). Spatial ability for STEM domains: Aligning over 50 years of cumulative psychological knowledge solidifies its importance. Journal of Educational Psychology, 101 , 817-835.

Ward, M., and Sweller, J. (1990). Structuring effective worked examples. Cognition and Instruction, 7 (1), 1-39.

Webb, R.M., Lubinski, D., and Benbow, C.P. (2007). Spatial ability: A neglected dimension in talent searches for intellectually precocious youth. Journal of Educational Psychology, 99 , 397-420.

Wheatley, G.H. (1984). Problem solving in school mathematics. MEPS Technical Report 84.01. West Lafayette, IN: Purdue University School Mathematics and Science Center.

Whitson, L., Bretz, S.L., and Towns, M.H. (2008). Characterizing the level of inquiry in the undergraduate laboratory. Journal of College Science Teaching, 37 (7), 52-58.

Winn, W. (1989). The design and use of instructional graphics. In H. Mandl and J.R. Levin (Eds.), Knowledge acquisition from text and pictures (pp. 125-144). Amsterdam: Elsevier.

Wu, H., and Shah, P. (2004). Exploring visuospatial thinking in chemistry learning. Science Education, 88 (3), 465-492.

Yerushalmi, E., Henderson, C., Heller, K., Heller, P., and Kuo, V. (2007). Physics faculty beliefs and values about the teaching and learning of problem solving. I: Mapping the common core. Physical Review Special Topics—Physics Education Research, 3 (2), 020109-1-020109-31.

Yildirim, T., Shuman, L., and Besterfield-Sacre, M. (2010). Model-eliciting activities: Assessing engineering student problem solving and skill integration processes. International Journal of Engineering Education, 26 (4), 831-845.

Adams, J.P., and Slater, T.F. (1998). Mysteries of the sky . Dubuque, IA: Kendall/Hunt.

Adams, J.P., and Slater, T.F. (2002). Learning through sharing: Supplementing the astronomy lecture with collaborative-learning group activities. Journal of College Science Teaching, 31 (6), 384-387.

Adams, J.P., Slater, T.F., Lindell-Adrian, R., Brissenden, G., and Wallace, J. (2002). Observations of student behavior in collaborative learning groups. Astronomy Education Review, 1 (1), 25-32.

Alexander, W.R. (2005). Assessment of teaching approaches in an introductory astronomy college classroom. Astronomy Education Review, 3 (2), 178-186.

Allen, D., and Tanner, K. (2009). Transformations: Approaches to college science teaching (1st ed.). New York: W.H. Freeman.

Anderson, W.L., Mitchell, S.M., and Osgood, M.P. (2008). Gauging the gaps in student problem-solving skills: Assessment of individual and group use of problem-solving strategies using online discussions. CBE-Life Sciences Education , 7 , 254-262.

Anderson, W.L., Sensibaugh, C.A., Osgood, M.P., and Mitchell, S.M. (2011). What really matters: Assessing individual problem-solving performance in the context of biological sciences. International Journal for the Scholarship of Teaching and Learning , 5 , 1.

Armstrong, N., Chang, S.M., and Brickman, M. (2007). Cooperative learning in industrial-sized biology classes. CBE-Life Sciences Education , 6 (2), 163-171.

Bailey, J.M., and Nagamine, K. (2009). Results from a case study investigation on the adoption of learner-centered strategies in introductory astronomy. Paper presented at the European Association for Research on Learning and Instruction Annual Meeting, Amsterdam, The Netherlands.

Barlow, A.E.L., and Villarejo, M. (2004). Making a difference for minorities: Evaluation of an educational enrichment program. Journal of Research in Science Teaching , 41 (9), 861-881.

Beatty, I.D., Gerace, W.J., Leonard, W.J., and Dufresne, R.J. (2006). Designing effective questions for classroom response system teaching. American Journal of Physics , 74 (1), 31-39.

Beichner, R.J. (1990). The effect of simultaneous motion presentation and graph generation in a kinematics lab. Journal of Research in Science Teaching , 27 (8), 803-815.

Beichner, R.J. (1996). The impact of video motion analysis on kinematics graph interpretation skills. American Journal of Physics , 64 (10), 1272-1277.

Beichner, R.J., Saul, J.M., Abbott, D.S., Morse, J.J., Deardorff, D.L., Allain, R.J., Bonham, S.W., Dancy, M.H., and Risley, J.S. (2007). The student-centered activities for large enrollment undergraduate programs (SCALE-UP) project. Reviews in PER Volume 1: Research-based reform of university physics. College Park, MD: American Association of Physics Teachers.

Bowen, C.W., and Phelps, A.J. (1997). Demonstration-based cooperative testing in general chemistry: A broader assessment-of-learning technique. Journal of Chemical Education , 74 (6), 715-719.

Boyle A.P., Maguire S., et al. (2004). Is fieldwork good? An analysis of the student view. Geological Society of America Abstracts with Programs , 36 (5), 156.

Brasell, H. (1987). The effect of real-time laboratory graphing on learning graphic representations of distance and velocity. Journal of Research in Science Teaching , 24 (4), 385-395.

Brewe, E. (2008). Modeling theory applied: Modeling instruction in introductory physics. American Journal of Physics , 76 (12), 1155-1160.

Brewe, E., Kramer, L., and O’Brien, G. (2009). Modeling instruction: Positive attitudinal shifts in introductory physics measured with CLASS. Physical Review Special Topics—Physics Education Research , 5 (1), 013102-1–013102-5.

Brickman, P., Gormally, C., Armstrong, N., and Hallar, B. (2009). Effects of inquiry-based learning on students’ science literacy skills and confidence. International Journal for the Scholarship of Teaching and Learning , 3 (2).

Brogt, E., Sabers, D., Prather, E.E., Deming, G.L., Hufnagel, B., and Slater, T.F. (2007). Analysis of the astronomy diagnostic test. Astronomy Education Review , 6 (1), 25-42.

Bruck, L.B., Towns, M., and Bretz, S.L. (2010). Faculty perspectives of undergraduate chemistry laboratory: Goals and obstacles to success. Journal of Chemical Education , 87 (12), 1416-1424.

Brungardt, J.B., and Zollman, D. (1995). Influence of interactive videodisc instruction using simultaneous-time analysis on kinematics graphing skills of high school physics students. Journal of Research in Science Teaching , 32 (8), 855-869.

Butler, R. (2008). Teaching geoscience through fieldwork, GEES subject centre learning and teaching guide . Plymouth, UK: Higher Education Academy Subject Centre for Geography, Earth and Environmental Science. Available: http://www.gees.ac.uk/pubs/guides/fw/fwgeosci.pdf .

Caldwell, J.E. (2007). Clickers in the large classroom: Current research and best-practice tips. CBE-Life Sciences Education , 6 (1), 9-20.

Catrambone, R., and Holyoak, K.J. (1989). Overcoming contextual limitations on problem-solving transfer. Journal of Experimental Psychology: Learning , Memory , and Cognition , 15 , 1147-1156.

Chasteen, S.V., and Pollock, S.J. (2008). Transforming upper-division electricity and magnetism. Journal of Research in Science Teaching , 21 , 221-233.

Clary, R.M., and Wandersee, J.H. (2007). A mixed methods analysis of the effects of an integrative geobiological study of petrified wood in introductory college geology classrooms. Journal of Research in Science Teaching , 44 (8), 1011-1035.

Cortright, R.N., Collins, H.L., Rodenbaugh, D.W., and DiCarlo, S.E. (2003). Student retention of course content is improved by collaborative-group testing. Advances in Physiology Education , 27 , 102-108.

Crouch, C.H., Fagen, A.P., Callan, J.P., and Mazur, E. (2004). Classroom demonstrations: Learning tools or entertainment? American Journal of Physics , 72 (6), 835-838.

Derting, T.L., and Ebert-May, D. (2010). Learner-centered inquiry in undergraduate biology: Positive relationships with long-term student achievement. CBE-Life Sciences Education , 9 (4), 462-472.

Dirks, C., and Cunningham, M. (2006). Enhancing diversity in science: Is teaching science process skills the answer? CBE-Life Sciences Education , 5 (3), 218-226.

Domin, D.S. (1999). A review of laboratory instruction styles. Journal of Chemical Education , 76 (4), 543-547.

Elby, A. (2001). Helping physics students learn how to learn. American Journal of Physics , 69 (7, Suppl. 1), S54-S64.

Elkins, J.T., and Elkins, N.M.L. (2007). Teaching geology in the field: Significant geoscience concept gains in entirely field-based introductory geology courses. Journal of Geoscience Education , 55 (2), 126-132.

Elliott, M.J., Stewart, K.K., and Lagowski, J.J. (2008). The role of the laboratory in chemistry instruction. Journal of Chemical Education , 85 (1), 145-149.

Etkina, E., and Van Heuvelen, A. (2007). Investigative science learning environment: A science-process approach to learning physics. In E.F. Redish and P.J. Cooney (Eds.), Research-based reform of university physics (vol. 1). College Park, MD: American Association of Physics Teachers. Available: http://www.per-central.org/document/ServeFile.cfm?ID=4988 .

Etkina, E., Gibbons, K., Holton, B.L., and Horton, G.K. (1999). Lessons learned: A case study of an integrated way of teaching introductory physics to at-risk students at Rutgers University. American Journal of Physics , 67 (9), 810-818.

Etkina, E., Van Heuvelen, A., White-Brahmia, S., Brookes, D.T., Gentile, M., Murthy, S., Rosengrant, D., and Warren, A. (2006). Scientific abilities and their assessment. Physical Review Special Topics—Physics Education Research , 2 (2), 020103-1–020103-15.

Etkina, E., Karelina, A., Ruibal-Villasenor, M., Rosengrant, D., Jordan, R., and Hmelo-Silver, C.E. (2010). Design and reflection help students develop scientific abilities: Learning in introductory physics laboratories. Journal of the Learning Sciences , 19 (1), 54-98.

Feisel, L.D., and Peterson, G.D. (2002). A colloquy on learning objectives for engineering education laboratories . Proceedings of the 2002 American Society for Engineering Education Annual Conference and Exposition. Washington, DC: American Society for Engineering Education.

Feisel, L.D., and Rosa, A.J. (2005). The role of the laboratory in undergraduate engineering education. Journal of Engineering Education , 94 (1), 121-130.

Finkelstein, N.D., Adams, W.K., Keller, C.J., Kohl, P.B., Perkins, K.K., Podolefsky, N.S., Reid, S., and LeMaster, R. (2005). When learning about the real world is better done virtually: A study of substituting computer simulations for laboratory equipment. Physical Review Special Topics—Physics Education Research , 1 (1), 010103-1–010103-8.

Formica, S.P., Easley, J.L., and Spraker, M.C. (2010). Transforming common-sense beliefs into Newtonian thinking through Just-in-Time Teaching. Physical Review Special Topics—Physics Education Research , 6 (2), 020106-1–020106-7.

Gabel, D. (1999). Improving teaching and learning through chemistry education research: A look to the future. Journal of Chemical Education , 76 (4), 548-554.

Gaffney, J.D.H., Richards, E., Kustusch, M.B., Ding, L., and Beichner, R.J. (2008). Scaling up education reform. Journal of College Science Teaching , 37 (5), 48-53.

Gentner, D., and Colhoun, J. (2010). Analogical processes in human thinking and learning. In B. Glatzeder, V. Goel, and A. V. Muller (Eds.), On thinking: Volume 2. Towards a theory of thinking (pp. 35-48). Berlin, Germany: Springer-Verlag.

Gick, M.L., and Holyoak, K.J. (1983). Schema induction and analogical transfer. Cognitive Psychology , 15 , 1-38.

Hake, R.R. (1998). Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. American Journal of Physics , 66 (1), 64-74.

Halme, D.G., Khodor, J., Mitchell, R., and Walker, G.C. (2006). A small-scale concept-based laboratory component: The best of both worlds. CBE-Life Sciences Education , 5 (1), 41-51.

Hanauer, D.I., Jacobs-Sera, D., Pedulla, M.L., Cresawn, S.G., Hendrix, R.W., and Hatfull, G.F. (2006). Teaching scientific inquiry. Science , 314 (5807), 1880-1881.

Handelsman, J., Miller, S., and Pfund, C. (2007). Scientific teaching . New York: Freeman.

Hart, C., Mulhall, P., Berry, A., Loughran, J., and Gunstone, R. (2000). What is the purpose of this experiment? Or can students learn something from doing experiments? Journal of Research in Science Teaching , 37 (7), 655-675.

Hatfull, G.F., Jacobs-Sera, D, Lawrence, J.G., Pope, W.H., Russell, D.A., Ko, C.C., Weber, R.J., Patel, M.C., Germane, K.L., Edgar, R.H., Hoyte, N.N., Bowman, C.A., Tantoco, A.T., Paladin, E.C., Myers, M.S., Smith, A.L., Grace, M.S., Pham, T.T., O’Brien, M.B., Vogelsberger, A.M., Hryckowian, A.J., Wynalek, J.L., Donis-Keller, H., Bogel, M.W., Peebles, C.L., Cresawn, S.G., and Hendrix, R.W. (2010). Comparative genomic analysis of sixty mycobacteriophage genomes: Genome clustering, gene acquisition and gene size. Journal of Molecular Biology , 397 (1), 119-143.

Hays, J.D., Pfirman, S., Blumenthal, M., Kastens, K., and Menke, W. (2000). Earth science instruction with digital data. Computers and the Geosciences , 26 , 657-668.

Herrington, D.G., and Nakhleh, M.B. (2003). What defines effective chemistry laboratory instruction? Teaching assistant and student perspectives. Journal of Chemical Education , 80 (10), 1197-1205.

Hofstein, A., and Lunetta, V.N. (2004). The laboratory in science education: Foundations for the twenty-first century. Science Education , 88 (1), 28-54.

Hofstein, A., and Mamlok-Naaman, R. (2007). The laboratory in science education: The state of the art. Chemistry Education: Research and Practice in Europe , 8 (2), 105-108.

Hollabaugh, M. (1995). Physics problem-solving in cooperative learning groups . Unpublished doctoral dissertation, University of Minnesota, Twin Cities. Available: http://groups.physics.umn.edu/physed/People/Hollabaugh%20Dissertation.pdf .

Hudgins, D.W., Prather, E.E., Grayson, D.J., and Smits, D.P. (2006). Effectiveness of collaborative ranking tasks on student understanding of key astronomy concepts. Astronomy Education Review , 5 (1), 1-22.

Hufnagel, B. (2002). Development of the Astronomy Diagnostic Test. Astronomy Education Review, 1 (1), 47-51.

Hufnagel, B., Slater, T.F., Deming, G.L., Adams, J.P., Adrien, R.L., Brick, C., and Zeilik, M. (2000). Pre-course results from the Astronomy Diagnostic Test. Publications of the Astronomical Society of Australia , 17 (2), 152-155.

Huntoon, J.E., Bluth, G.J.S., and Kennedy, W.A. (2001). Measuring the effects of a research-based field experience on undergraduates and K-12 teachers. Journal of Geoscience Education , 49 (3), 235-248.

Jacoby, L.L. (1978). On interpreting the effects of repetition: Solving a problem versus remembering a solution. Journal of Verbal Learning and Verbal Behavior , 17 , 649-667.

Jalil, P.A. (2006). A procedural problem in laboratory teaching: Experiment and explain, or vice-versa? Journal of Chemical Education , 83 (1), 159-163.

Johnson, D.W., Johnson, R.T., and Holubec, E. (1990). Circles of learning: Cooperation in the classroom . Edina, MN: Interaction Book.

Johnson, D.W., Johnson, R.T., and Smith, K.A. (1991). Cooperative learning: Increasing college faculty instructional productivity. ASHE-ERIC Higher Education Report No. 4. Washington, DC: George Washington University School of Education and Human Development.

Johnson, D.W., Johnson, R.T., and Smith K.A. (1998). Cooperative learning returns to college: What evidence is there that it works? Change , 30 , 26-35.

Johnson, D.W., Johnson, R.T., and Smith, K.A. (2007). The state of cooperative learning in postsecondary and professional settings. Educational Psychology Review , 19 (1), 15-29.

Johnson, D.W., Johnson, R.T., and Smith, K.A. (2011). Lecturing with informal cooperative learning groups. In J. Cooper and P. Robinson (Eds.), Small group learning in higher education: Research and practice . Oklahoma City, OK: New Forums Press.

Johnson, M.A., and Lawson, A.E. (1998). What are the relative effects of reasoning ability and prior knowledge on biology achievement in expository and inquiry classes? Journal of Research in Science Teaching , 35 (1), 89-103.

Kanter, D.E., Smith, H.D., McKenna, A., Rieger, C., and Linsenmeier, R.A. (2003). Inquiry-based laboratory instruction throws out the cookbook and improves learning . Evanston, IL: Northwestern University.

Keller, C., Finkelstein, N.D., Perkins, K.K., Pollock, S.J., Turpen, C., and Dubson, M. (2007). Research-based practices for effective clicker use. AIP Conference Proceedings , 951 , 128-131. Available: http://www.compadre.org/Repository/document/servefile.cfm?ID=9085&docid=2003 .

Kirschner, P.A., and Meester, M.A.M. (1988). The laboratory in higher science education: Problems, premises and objectives. Higher Education , 17 (1), 81-98.

Knight, J.K., and Wood, W.B. (2005). Teaching more by lecturing less. Cell Biology Education , 4 (4), 298-310.

Kortz, K.M., Smay, J.J., and Murray, D.P. (2008). Increasing learning in introductory geoscience courses using lecture tutorials. Journal of Geoscience Education , 56 (3), 280-290.

Lasry, N. (2008). Clickers or flashcards: Is there really a difference? Physics Teacher , 46 (4), 242-244.

Laws, P.W. (1991). Calculus-based physics without lectures. Physics Today , 44 (12), 24-31.

Laws, P.W. (2004). A unit on oscillations: Determinism and chaos for introductory physics students. American Journal of Physics, 72 (4).

Laws, P.W., Rosborough, P.J., and Poodry, F.J. (1999). Women’s responses to an activity-based introductory physics program. American Journal of Physics , 67 (7, Suppl. 1), S32-S37.

Lawson, A.E. (1988). A better way to teach biology. American Biology Teacher , 50 (5), 266-274.

Lazarowitz, R., and Tamir, P. (1994). Research on using laboratory instruction in science. In D.L. Gabel (Ed.), Handbook of research on science teaching and learning: A project of the National Science Teachers Association (pp. 94-128). New York: Macmillan.

Len, P.M. (2007). Different reward structures to motivate student interaction with electronic response systems in astronomy. Astronomy Education Review , 5 (2), 5-15.

Linneman, S., and Plake, T. (2006). Searching for the difference: A controlled test of just-in-time teaching for large-enrollment introductory geology courses. Journal of Geoscience Education , 54 (1), 18-24.

Lipshitz, R., Klein, G., Orasanu, J., and Salas, E. (2001). Focus article: Taking stock of naturalistic decision making. Journal of Behavioral Decision Making , 14 (5), 331-352.

Lopresto, M.C. (2010). Comparing a lecture with a tutorial in introductory astronomy. Physics Education , 45 (2), 196-201.

Lopresto, M.C., and Murrell, S.R. (2009). Using the Star Properties Concept Inventory to compare instruction with lecture tutorials to traditional lectures. Astronomy Education Review , 8 (1), 010105-1–010105-5.

Lord, T., and Orkwiszewski, T. (2006). Moving from didactic to inquiry-based instruction in a science laboratory. American Biology Teacher , 68 (6), 342-345.

Luo, W. (2008). Just-in-Time-Teaching (JITT) improves students’ performance in classes: Adaptation of JITT in four geography courses. Journal of Geoscience Education , 56 (2), 166-171.

Lynch, D.R., Russell, J.S., Evans, J.C., and Sutterer, K.G. (2009). Beyond the cognitive: The affective domain, values, and the achievement of the vision. Journal of Professional Issues in Engineering Education and Practice , 135 (1), 47-56.

Lyons, D.L. (2011). Impact of backwards faded scaffolding approach to inquiry-based astronomy laboratory experiences on undergraduate non-science majors’ views of scientific inquiry . Ph.D. Dissertation, University of Wyoming.

MacArthur, J.R., and Jones, L.L. (2008). A review of literature reports of clickers applicable to college chemistry classrooms. Chemistry Education Research and Practice , 9 (3), 187-195.

Malina, E.G., and Nakhleh, M.B. (2003). How students use scientific instruments to create understanding: CCD spectrophotometers. Journal of Chemical Education , 80 (6), 691-698.

Marrs, K.A., and Novak, G. (2004). Just-in-time teaching in biology: Creating an active learner classroom using the internet. Cell Biology Education , 3 (1), 049-061.

Matsui, J., Liu, R., and Kane, C.M. (2003). Evaluating a science diversity program at UC Berkeley: More questions than answers. Cell Biology Education , 2 (2), 117-121.

Mayer, R.E. (2010). Applying the science of learning . Upper Saddle River, NJ: Pearson.

McConnell, D.A., Steer, D.N., and Owens, K.D. (2003). Assessment and active learning strategies for introductory geology courses. Journal of Geoscience Education , 51 (2), 205-216.

McConnell, D.A., Steer, D.N., Owens, K., Borowski, W., Dick, J., Foos, A., Knott, J.R., Malone, M., McGrew, H., Van Horn, S., Greer, L., and Heaney, P.J. (2006). Using conceptests to assess and improve student conceptual understanding in introductory geoscience courses. Journal of Geoscience Education , 54 (1), 61-68.

McDermott, L.C., and Shaffer, P.S. (2002). Tutorials in introductory physics (1st ed.). Upper Saddle River, NJ: Prentice Hall.

McDermott, L.C., and the Physics Education Group at the University of Washington. (1996a). Physics by inquiry: An introduction to physics and the physical sciences (vol. I). New York: John Wiley & Sons.

McDermott, L.C., and the Physics Education Group at the University of Washington. (1996b). Physics by inquiry: An introduction to physics and the physical sciences (vol. II). New York: John Wiley & Sons.

Mestre, J.P., Ross, B.H., Brookes, D.T., Smith, A.D., and Nokes, T.J. (2009). How cognitive science can promote conceptual understanding in physics classrooms. In I.M. Saleh and M.S. Khine (Eds.), Fostering scientific habits of mind: Pedagogical knowledge and best practices in science education (pp. 3-8). Rotterdam, The Netherlands: Sense.

Miller, L.S., Nakhleh, M.B., Nash, J.J., and Meyer, J.A. (2004). Students’ attitudes toward and conceptual understanding of chemical instrumentation. Journal of Chemical Education , 81 (12), 1801-1808.

Moravec, M., Williams, A., Aguilar-Roca, N., and O’Dowd, D.K. (2010). Learn before lecture: A strategy that improves learning outcomes in a large introductory biology class. CBE-Life Sciences Education , 9 (4), 473-481.

Nelson, K.G., Huysken, K., and Kilibarda, Z. (2010). Assessing the impact of geoscience laboratories on student learning: Who benefits from introductory labs? Journal of Geoscience Education , 58 (1), 43-50.

Novak, G.M. (1999). Just-in-time teaching: Blending active learning with web technology . Upper Saddle River, NJ: Prentice Hall.

Novick, L.R., and Holyoak, K.J. (1991). Mathematical problem solving by analogy. Journal of Experimental Psychology: Learning , Memory , and Cognition , 17 , 398-415.

Orgill, M., and Bodner, G.M. (2004). What research tells us about using analogies to teach chemistry. Chemistry Education Research and Practice , 5 (1), 15-32.

Orgill, M., and Bodner, G.M. (2007). Locks and keys: An analysis of biochemistry students’ use of analogies. Biochemistry and Molecular Biology Education , 35 (4), 244-254.

Piburn, M., van der Hoeven Kraft, K., and Pacheco, H. (2011). A new century for geoscience education research . Paper presented at the Second Committee Meeting on the Status, Contributions, and Future Directions of Discipline-Based Education Research. Available: http://www7.nationalacademies.org/bose/DBER_Piburn_October_Paper.pdf .

Podolefsky, N.S., and Finkelstein, N.D. (2006). The use of analogy in learning physics: The role of representations. Physics Review Special Topics—Physics Education Research, 2, 020101. Available: http://prst-per.aps.org/abstract/PRSTPER/v2/i2/e020101 .

Podolefsky, N.S., and Finkelstein, N.D. (2007a). Analogical scaffolding and the learning of abstract ideas in physics: Empirical studies. Physics Review Special Topics—Physics Education Research, 3 , 020104. Available: http://prst-er.aps.org/abstract/PRSTPER/v3/i2/e020104 .

Podolefsky, N.S., and Finkelstein, N.D. (2007b). Analogical scaffolding and the learning of abstract ideas in physics: An example from electromagnetic waves. Physics Review Special Topics—Physics Education Research, 3 , 010109.

Prather, E.E., Slater, T.F., Adams, J., and Brissenden, B. (2007). Lecture-tutorials for introductory astronomy (2nd ed.). New York: Prentice Hall.

Prince, M. (2004). Does active learning work? A review of the research. Journal of Engineering Education , 93 (3), 223-231.

Pyle, E.J. (2009). The evaluation of field course experiences: A framework for development, improvement, and reporting. Geological Society of America Special Papers , 461 , 341-356.

Rao, S.P., Collins, H.L., and DiCarlo, S.E. (2002). Collaborative testing enhances student learning. American Journal of Physiology—Advances in Physiology Education , 26 (1-4), 37-41.

Rath, K.A., Peterfreund, A.R., Xenos, S.P., Bayliss, F., and Carnal, N. (2007). Supplemental instruction in Introductory Biology I: Enhancing the performance and retention of under-represented minority students. CBE-Life Sciences Education, 6 (3), 203-216.

Redish, E.F., Steinberg, R.N., and Saul, J.M. (1998). Student expectations in introductory physics. American Journal of Physics , 66 , 212-224.

Riggs, E.M., Balliet, R., and Lieder, C.C. (2009). Using GPS tracking to understand problem solving during geologic field examinations. Geological Society of America Special Papers, 461 .

Rissing, S.W., and Cogan, J.G. (2009). Can an inquiry approach improve college student learning in a teaching laboratory? CBE-Life Sciences Education , 8 (1), 55-61.

Ross, B.H., and Kennedy, P.T. (1990). Generalizing from the use of earlier examples in problem solving. Journal of Experimental Psychology: Learning , Memory , and Cognition , 16 , 42-55.

Rudd, J.A., Greenbowe, T.J., and Hand, B.M. (2007). Using the science writing heuristic to improve students’ understanding of general equilibrium. Journal of Chemical Education , 84 (12), 2007-2011.

Ruiz-Primo, M.A., Briggs, D., Iverson, H., Talbot, R., and Shepard, L.A. (2011). Impact of undergraduate science course innovations on learning. Science , 331 (6022), 1269-1270.

Sandi-Urena, S., Cooper, M., Gatlin, T. and Bhattacharyya, G. (2011). Students’ experience in a general chemistry cooperative problem-based laboratory. Chemistry Education Research and Practice , 12 , 434-442.

Sandi-Urena, S., Cooper, M., Gatlin, T., Bhattacharyya, G., and Stevens, R. (in press). Effect of cooperative problem-based lab instruction on metacognition and problem-solving skills. Journal of Chemical Education .

Schwartz, D.L., and Bransford, J.D. (1998). A time for telling. Cognition and Instruction , 16 (4), 475-522.

Seymour, E., and Hewitt, N. (1997). Talking about leaving: Why undergraduates leave the sciences . Boulder, CO: Westview Press.

Seymour, E., Hunter, A.B., Laursen, S.L., and Deantoni, T. (2004). Establishing the benefits of research experiences for undergraduates in the sciences: First findings from a three-year study. Science Education , 88 (4), 493-534.

Shaffer, C.D., Alvarez, C., Bailey, C., Barnard, D., Bhalla, S., Chandrasekaran, C., Chandrasekaran, V., Chung, H.M., Dorer, D.R., Du, C., Eckdahl, T.T., Poet, J.L., Frohlich, D., Goodman, A.L., Gosser, Y., Hauser, C., Hoopes, L.L., Johnson, D., Jones, C.J., Kaehler, M., Kokan, N., Kopp, O.R., Kuleck, G.A., McNeil, G., Moss, R., Myka, J.L., Nagengast, A., Morris, R., Overvoorde, P.J., Shoop, E., Parrish, S., Reed, K., Regisford, E.G., Revie, D., Rosenwald, A.G., Saville, K., Schroeder, S., Shaw, M., Skuse, G., Smith, C., Smith, M., Spana, E.P., Spratt, M., Stamm, J., Thompson, J.S., Wawersik, M., Wilson, B.A., Youngblom, J., Leung, W., Buhler, J., Mardis, E.R., Lopatto, D., and Elgin, S.C. (2010). The genomics education partnership: Successful integration of research into laboratory classes at a diverse group of undergraduate institutions. CBE-Life Sciences Education, 9 (1), 55-69.

Sibbernsen, K.J. (2010). The impact of collaborative groups versus individuals in undergraduate inquiry-based astronomy laboratory learning experiences . Ph.D. Dissertation, Capella University. Available: http://www.docstoc.com/docs/77505821/The-impact-of-collaborative-groups-versus-individuals-in-undergraduate-inquiry-based-astronomy-laboratory-learning-exercises .

Simmons, M.E., Wu, X.B., Knight, S.L., and Lopez, R.R. (2008). Assessing the influence of field- and GIS-based inquiry on student attitude and conceptual knowledge in an undergraduate ecology lab. CBE-Life Sciences Education , 7 (3), 338-345.

Skala, C., Slater, T.F., and Adams, J.P. (2000). Qualitative analysis of collaborative learning groups in large enrollment introductory astronomy. Publications of the Astronomical Society of Australia , 17 (2), 185-193.

Slater, S.J., Slater, T.F., and Shaner, A. (2008). Impact of backwards faded scaffolding in an astronomy course for pre-service elementary teachers based on inquiry. Journal of Geoscience Education , 56 (5), 408-416.

Slater, S.J., Slater, T.F., and Lyons, D. (2010). Engaging in astronomical inquiry . New York: W.H. Freeman.

Slavin, R.E., Hurley, E.A., and Chamberlain, A. (2003). Cooperative learning and achievement: Theory and practice. In W.M. Reynolds and G.E. Miller (Eds.), Handbook of psychology (vol. 7, pp. 177-198). New York: John Wiley & Sons.

Smith, K.A. (2000). Going deeper: Formal small-group learning in large classes. In J. Macgregor, J. Cooper, K. Smith, and P. Robinson (Eds.), Strategies for energizing large classes: From small groups to learning communities: New directions for teaching and learning (pp. 25-46). San Francisco, CA: Jossey-Bass.

Smith, K.A., Sheppard, S.D., Johnson, D.W., and Johnson, R.T. (2005). Pedagogies of engagement: Classroom-based practices. Journal of Engineering Education , 94 (1), 87-100.

Smith, M.K., Wood, W.B., Adams, W.K., Wieman, C., Knight, J.K., Guild, N., and Su, T.T. (2009). Why peer discussion improves student performance on in-class concept questions. Science , 323 (5910), 122-124.

Smith, M.K., Wood, W.B., Krauter, K., and Knight, J.K. (2011). Combining peer discussion with instructor explanation increases student learning from in-class concept questions. CBE-Life Sciences Education , 10 (1), 55-63.

Sokoloff, D.R., and Thornton R.K. (2004). Interactive lecture demonstrations: Active learning in introductory physics . New York: John Wiley & Sons.

Sorensen, C.M., Churukian, A.D., Maleki, S., and Zollman, D.A. (2006). The new studio format for instruction of introductory physics. American Journal of Physics , 74 (12), 1077-1082.

Springer, L., Stanne, M.E., and Donovan, S.S. (1999). Effects of small-group learning on undergraduates in science, mathematics, engineering, and technology: A meta-analysis. Review of Educational Research , 69 (1), 21-51.

Steinberg, R.N., Wittmann, M.C., and Redish, E.F. (1997). Mathematical tutorials in introductory physics. AIP Conference Proceedings, 399, 1075-1092.

Stelzer, T., Gladding, G., Mestre, J.P., and Brookes, D.T. (2009). Comparing the efficacy of multimedia modules with traditional textbooks for learning introductory physics content. American Journal of Physics , 77 (2), 184-190.

Stokes, A., and Boyle, A. (2009). The undergraduate geoscience fieldwork experience: Influencing factors and implications for learning. Geological Society of America Special Papers , 461 , 291-311.

Thacker, B., Eunsook, K., Kelvin, T., and Lea, S.M. (1994). Comparing problem-solving performance of physics students in inquiry-based and traditional introductory physics courses. American Journal of Physics , 62 (7), 627-633.

Tien, L.T., Roth, V., and Kampmeier, J.A. (2002). Implementation of a peer-led team learning instructional approach in an undergraduate organic chemistry course. Journal of Research in Science Teaching , 39 (7), 606-632.

Towns, M., and Kraft, A. (2011). Review and synthesis of research in chemical education from 2000-2010 . Paper presented at the Second Committee Meeting on the Status, Contributions, and Future Directions of Discipline-Based Education Research. Available: http://www7.nationalacademies.org/bose/DBER_Towns_October_Paper.pdf .

Vygotsky, L.S. (1978). Mind in society: The development of higher psychological processes . Cambridge, MA: Harvard University Press.

White, R.T. (1996). The link between the laboratory and learning. International Journal of Science Education , 18 (7), 761-774.

Whitmeyer, S., Mogk, D., and Pyle, E. (Eds). (2009). Field geology education: Historical perspectives and modern approaches . Boulder, CO: Geological Society of America.

Whitson, L., Raker, J., Bretz, S.L., and Towns, M.H. (2008). Characterizing the level of inquiry in the undergraduate laboratory. Journal of College Science Teaching, 37 (7), 52-58.

Wieman, C., Perkins, K., Gilbert, S., Benay, F., Kennedy, S., Semsar, K., Knight, J., Shi, J., Smith, M., Kelly, T., Taylor, J., Yurk, H., Birol, G., Langdon, L., Pentecost, T., Stewart, J., Arthurs, L., Bair, A., Stempien, J., Gilley, B., Jones, F., Kennedy, B., Chasteen, S., and Simon, B. (2008). Clicker resource guide: An instructor’s guide to the effective use of personal response systems (clickers) in teaching . Available: http://www.colorado.edu/sei/documents/clickeruse_guide0108.pdf .

Wittmann, M.C., Steinberg, R., and Redish, E. (2004). Activity-based tutorials. Volume 1: Introductory physics . Hoboken, NJ: John Wiley & Sons.

Wittmann, M.C., Steinberg, R., and Redish, E. (2005). Activity-based tutorials. Volume 2: Modern physics . Hoboken, NJ: John Wiley & Sons.

Wood, W.B. (2004). Clickers: A teaching gimmick that works. Developmental Cell , 7 (6), 796-798.

Wood, W.B. (2009). Innovations in teaching undergraduate biology and why we need them. Annual Review of Cell and Developmental Biology , 25 , 93-112.

Wright, R., and Boggs, J. (2002). Learning cell biology as a team: A project-based approach to upper-division cell biology. Cell Biology Education , 1 (4), 145-153.

Yerushalmi, E., Henderson, C., Heller, K., Heller, P., and Kuo, V. (2007). Physics faculty beliefs and values about the teaching and learning of problem solving. I. Mapping the common core. Physical Review Special Topics—Physics Education Research , 3 (2), 020109-1–020109-31.

Yuretich, R.F., Khan, S.A., Leckie, R.M., and Clement, J.J. (2001). Active-learning methods to improve student performance and scientific interest in a large introductory oceanography course. Journal of Geoscience Education , 49 (2), 111-119.

Zsambok, C., and Klein, G. (1997). Naturalistic decision making . Mahwah, NJ: Lawrence Erlbaum Associates.

Adams, W.K., Perkins, K.K., Podolefsky, N.S., Dubson, M., Finkelstein, N.D., and Wieman, C.E. (2006). New instrument for measuring student beliefs about physics and learning physics: The Colorado Learning Attitudes about Science Survey. Physical Review Special Topics—Physics Education Research , 2 (1).

Ainsworth, S., and Loizou, A.T. (2003). The effects of self-explaining when learning with text or diagrams. Cognitive Science , 27, 669-681.

Ambrose, S.A., Bridges, M.W., DiPietro, M., Lovett, M.C., Norman, M.K., and Mayer, R.E. (2010). How learning works: Seven research-based principles for smart teaching . San Francisco, CA: Jossey-Bass.

American Association of Physics Teachers. (1997). Goals of the introductory physics laboratory. The Physics Teacher, 35 , 546-548.

Atman, C., Sheppard, S.D., Turns, J., Adams, R.S., Fleming, L.N., Stevens, R., Streveler, R.A., Smith, K.A., Miller, R.L., Leifer, L.J., Yasuhara, K., and Lund, D. (2010). Enabling engineering student success: The final report for the Center for the Advancement of Engineering Education. Technical Report CAEE-TR-10-02. San Rafael, CA: Morgan & Claypool.

Ausubel, D.P. (1968). Educational psychology , a cognitive view . New York: Holt, Rinehart and Winston.

Ausubel, D.P., Novak, J.D., and Hanesian, H. (1978). Educational psychology: A cognitive view (2d ed.). New York: Holt, Rinehart and Winston.

Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory . Englewood Cliffs, NJ: Prentice-Hall.

Barbera, J., Ams, W.K., Wieman, C.E., and Perkins, K.K. (2008). Modifying and validating the Colorado Learning Attitudes about Science Survey for use in chemistry. Journal of Chemical Education, 85 , 1435-1439.

Bassok, M. (1990). Transfer of domain-specific problem-solving procedures. Journal of Experimental Psychology: Learning , Memory , and Cognition , 16 (3), 522-533.

Bassok, M. (2003). Analogical transfer in problem solving. In J.E. Davidson and R.J. Sternberg (Eds.), Psychology of problem solving (pp. 343-369). New York: Cambridge University Press.

Bassok, M., and Holyoak, K.J. (1989). Interdomain transfer between isomorphic topics in algebra and physics. Journal of Experimental Psychology: Learning , Memory , and Cognition , 15 (1), 153-166.

Boyle, A., Maguire, S., Martin, A., Milsom, C., Nash, R., Rawlinson, S., Turner, A., Wurthmann, S., and Conchie, S. (2007). Fieldwork is good: The student perception and the affective domain. Journal of Geography in Higher Education , 31 (2), 299-317.

Brandriet, A.R., Xu, X., Bretz, S.L., and Lewis, J.E. (2011). Diagnosing changes in attitude in first-year college chemistry students with a shortened version of Bauer’s semantic differential. Chemistry Education Research and Practice , 12 (2), 271-278.

Bransford, J.D., and Schwartz, D.L. (1999). Rethinking transfer: A simple proposal with multiple implications. Review of Research in Education , 24 , 61-100.

Bretz, S.L. (2001). Novak’s theory of education: Human constructivism and meaningful learning. Journal of Chemical Education , 78 (8), 1107.

Brown, A.L. (1978). Knowing when, where, and how to remember: A problem of metacognition. In R. Glaser (Ed.), Advances in instructional psychology (vol. 2, pp. 77-165). Hillsdale, NJ: Lawrence Erlbaum Associates.

Catrambone, R. (1998). The subgoal learning model: Creating better examples so that students can solve novel problems. Journal of Experimental Psychology: General, 127 , 355-376.

Chi, M.T.H., and Bassok, M. (1989). Learning from examples via self-explanations. In L.B. Resnick (Ed.), Knowing , learning , and instruction: Essays in honor of Robert Glaser (pp. 251-282). Hillsdale, NJ: Lawrence Erlbaum Associates.

Chi, M.T.H., and VanLehn, K. (1991). The content of physics self-explanations. Journal of the Learning Sciences , 1 (1), 69-106.

Clark, I.F., and James, P.R. (2004). Using concept maps to plan an introductory structural geology course. Journal of Geoscience Education , 52 (3), 224-230.

Coil, D., Wenderoth, M.P., Cunningham, M., and Dirks, C. (2010). Teaching the process of science: Faculty perceptions and an effective methodology. CBE-Life Sciences Education , 9 (4), 524-535.

Collins, H., and Pinch, T. (1993). The Golem: What everyone should know about science . Cambridge, MA: Cambridge University Press.

Connor, C. (2009). Field glaciology and earth systems science: The Juneau Icefield Research Program (JIRP), 1946-2008. Geological Society of America Special Papers , 461 , 173-184.

Cooper, M.M., and Sandi-Urena, S. (2009). Design and validation of an instrument to assess metacognitive skillfulness in chemistry problem solving. Journal of Chemical Education , 86 (2), 240-245.

Cooper, M.M., Sandi-Urena, S., and Stevens, R. (2008). Reliable multimethod assessment of metacognition use in chemistry problem solving. Chemistry Education Research and Practice , 9 (1), 18-24.

Cox, A.J., and Junkin, W.F., III. (2002). Enhanced student learning in the introductory physics laboratory. Physics Education , 37 (1), 37-44.

Craik, F.I.M, and Lockhart, R.S. (1972). Levels of processing: A framework for memory research. Journal of Verbal Learning and Verbal Behavior, 11 , 671-684.

Crompton, J.L., and Sellar, C. (1981). Do outdoor education experiences contribute to positive development in the affective domain? Journal of Environmental Education , 12 (4), 21-29.

Cummings, K., Marx, J., Thornton, R., and Kuhl, D. (1999). Evaluating innovation in studio physics. American Journal of Physics, 67 (7, Suppl. 1), S38-S44.

De Leone, C.J., and Gire, E. (2006). Is instructional emphasis on the use of non-mathematical representations worth the effort? AIP Conference Proceedings , 818 , 45-48.

DeBoer, G.E. (1991). A history of ideas in science education . New York: Teachers College Press.

Dunbar, K. (1995). How scientists really reason: Scientific reasoning in real-world laboratories. In R.J. Sternberg and J.E. Davidson (Eds.), Mechanisms of insight (pp. 365-395). Cambridge, MA: MIT Press.

Elkins, J., Elkins, N.M.L., and Hemmings, S.N.J. (2008). GeoJourney: A nine-week introductory field program with an interdisciplinary approach to teaching geology, Native American cultures, and environmental studies. Journal of College Science Teaching , 37 (3), 18-28.

Flavell, J.H. (1979). Metacognition and cognitive monitoring: A new area of cognitive-developmental inquiry. American Psychologist , 34 (10), 906-911.

Flick, L., and Lederman, N.G. (Eds.). (2004). Scientific inquiry and nature of science: Implications for teaching, learning, and teacher education . The Netherlands: Kluwer Academic.

Fuller, I., Gaskin, S., and Scott, I. (2003). Student perceptions of geography and environmental science fieldwork in the light of restricted access to the field, caused by foot and mouth disease in the UK in 2001. Journal of Geography In Higher Education , 27 (1), 79-102.

Gawel, J.E., and Greengrove, C.L. (2005). Designing undergraduate research experiences for nontraditional student learning at sea. Journal of Geoscience Education , 53 , 31-36.

Gilligan, M.R., Verity, P.G., Cook, C.B., Cook, S.B., Booth, M.G., and Frischer, M.E. (2007). Building a diverse and innovative ocean workforce through collaboration and partnerships that integrate research and education: HBCUs and marine laboratories . Available: http://www.skio.usg.edu/publications/downloads/pdfs/_pubs/GilliganEA07.pdf .

Glynn, S.M., and Koballa, T.R. (2006). Motivation to learn in college science. In J. Mintzes and W.H. Leonard (Eds.), Handbook of college science teaching (pp. 25-32). Arlington, VA: National Science Teachers Association Press.

Goertzen, R.M., Scherr, R.E., and Elby, A. (2009). Accounting for tutorial teaching assistants’ buy-in to reform instruction. Physical Review Special Topics—Physics Education Research, 5 (2), 020109.

Goertzen, R.M., Scherr, R.E., and Elby, A. (2010). Respecting tutorial instructors’ beliefs and experiences: A case study of a physics teaching assistant. Physical Review Special Topics—Physics Education Research , 6 (2), 020125-1–020125-11.

Gray, J.R. (2004). Integration of emotion and cognitive control. Current Directions in Psychological Science , 13 (2), 46-48.

Gregerman, S.R. (2008). The role of undergraduate research in student retention , academic engagement , and the pursuit of graduate education. Paper presented at the National Research Council’s Workshop on Linking Evidence to Promising Practices in STEM Undergraduate Education, Washington, DC. Available: http://www7.nationalacademies.org/bose/Gregerman_CommissionedPaper.pdf .

Grove, N., and Bretz, S.L. (2007). CHEMX: An instrument to assess students’ cognitive expectations for learning chemistry. Journal of Chemical Education , 84 (9), 1524-1529.

Gunter, M.E. (2004). The polarized light microscope: Should we teach the use of a 19th century instrument in the 21st century? Journal of Geoscience Education , 52 (1), 34-44.

Halloun, I. (1997). Views about science and physics achievement: The VASS story. AIP Conference Proceedings, 399, 605-614.

Hammer, D. (1994). Epistemological beliefs in introductory physics. Cognition and Instruction , 12 (2), 151-183.

Hammer, D. (1995). Epistemological considerations in teaching introductory physics. Science Education , 79 (4), 393-413.

Hammer, D., Elby, A., Scherr, R., and Redish, E.F. (2005). Resources, framing, and transfer. In J. Mestre (Ed.), Transfer of learning from a modern multidisciplinary perspective (pp. 89-119). Greenwich, CT: Information Age.

Henderson, C., and Dancy, M.H. (2007). Barriers to the use of research-based instructional strategies: The influence of both individual and situational characteristics. Physical Review Special Topics—Physics Education Research , 3 (2), 020102-1–020102-14.

Henderson, C., Yerushalmi, E., Kuo, V.H., Heller, P., and Heller, K. (2004). Grading student problem solutions: The challenge of sending a consistent message. American Journal of Physics , 72 (2), 164-169.

Henderson, C., Yerushalmi, E., Kuo, V.H., Heller, K., and Heller, P. (2007). Physics faculty beliefs and values about the teaching and learning of problem solving II: Procedures for measurement and analysis. Physical Review Special Topics—Physics Education Research , 3 (2), 020110-1–020110-12.

Hilgard, E.R. (2006). The trilogy of mind: Cognition, affection and conation. Journal of the History of the Behavioral Sciences, 16 (2), 107-117.

Holyoak, K.J., and Koh, K. (1987). Surface and structural similarity in analogical transfer. Memory and Cognition , 15 (4), 332-340.

Hunter, A., Laursen, S., Seymour, E. (2007). Becoming a scientist: The role of undergraduate research in students’ cognitive, personal and professional development. Science Education , 91 (1), 36-74.

Jarrett, O.S., and Burnley, P.C. (2010). Lessons on the role of fun/playfulness from a geology undergraduate summer research program. Journal of Geoscience Education , 58 (2), 110-120.

Kaminsky, J.A., and Sloutsky, V.M. (2012). Representation and transfer of abstract mathematical concepts in adolescence and young adulthood. In V.F. Reyna, S.B. Chapman, M.R. Dougherty, and J. Confrey (Eds.), The adolescent brain: Learning, reasoning, and decision making (pp. 67-93). Washington, DC: American Psychological Association.

Karabinos, P., Stoll, H.M., and Fox, W.T. (1992). Attracting students to science through field exercises in introductory geology courses. Journal of Geological Education , 40 , 302-305.

Kardash, C.M. (2000). Evaluation of an undergraduate research experience: Perceptions of undergraduate interns and their faculty mentors. Journal of Educational Psychology , 92 (1), 191-201.

Karelina, A., and Etkina, E. (2007). Acting like a physicist: Student approach study to experimental design. Physical Review Special Topics—Physics Education Research, 3.

Karpicke, J.D., Butler, A.C., and Roediger, H.L. (2009). Metacognitive strategies in student learning: Do students practice retrieval when they study on their own? Memory, 17 , 471-479.

Karplus, R. (1977). Science teaching and the development of reasoning. Journal of Research in Science Teaching , 14 , 169-175.

Kelly, R.J.L. (2007). Exploring how different features of animations of sodium chloride dissolution affect students’ explanations. Journal of Science Education and Technology , 16 (5), 413-429.

Kempa, R.F., and Orion, N. (1996). Students’ perception of co-operative learning in earth science fieldwork: Research in Science and Technological Education, 14 (1), 33-41.

Kitchen, E., Bell, J.D., Reeve, S., Sudweeks, R.R., and Bradshaw, W.S. (2003). Teaching cell biology in the large-enrollment classroom: Methods to promote analytical thinking and assessment of their effectiveness. Cell Biology Education , 2 (3), 180-194.

Kost-Smith, L.E., Pollock, S.J., and Finkelstein, N.D. (2010). Gender disparities in second-semester college physics: The incremental effects of a smog of bias. Physical Review Special Topics—Physics Education Research , 6 (2). Available: http://www.colorado.edu/physics/EducationIssues/papers/Kost_etal/gender_disparities.pdf .

Krathwohl, D.R., Bloom, B.S., and Masia, B.B. (1964). Taxonomy of educational objectives: Handbook II: Affective domain . New York: David McKay.

Lave, J., and Wenger, E. (1991). Situated learning: Legitimate peripheral participation . Cambridge, UK: Cambridge University Press.

Lin, X., Schwartz, D.L., and Hatano, G. (2005). Toward teachers’ adaptive metacognition. Educational Psychologist, 40 (4), 245-255.

Locks, A.M., and Gregerman, S.R. (2008). Undergraduate research as an institutional retention strategy: The University of Michigan model. In R. Taraban and R.L. Blanton (Eds.), Creating effective undergraduate research programs in science: The transformation from student to scientist (pp. 11-32). New York: Teachers College Press.

Lopatto, D. (2007). Undergraduate research experiences support science career decisions and active learning. CBE-Life Sciences Education , 6 (4), 297-306.

Lopatto, D. (2010). Science in solution: The impact of undergraduate research on student learning (1st ed.). Washington, DC: Council on Undergraduate Research.

Lyons, D.L. (2011). Impact of backwards faded scaffolding approach to inquiry-based astronomy laboratory experiences on undergraduate non-science majors’ views of scientific inquiry . Ph.D. dissertation, University of Wyoming.

Maguire, S. (1998). Gender differences in attitudes to undergraduate fieldwork. Area , 30 (3), 207-214.

Manner, B.M. (1995). Field studies benefit students and teachers. Journal of Geological Education , 43 , 128-131.

Marques, L., Praia, J., and Kempa, R. (2003). A study of students’ perceptions of the organization and effectiveness of fieldwork in earth sciences education. Research in Science and Technological Education , 21 , 265-278.

Mattox, A.C., Reisner, B.A., and Rickey, D. (2006). What happens when chemical compounds are added to water? An introduction to the Model-Observe-Reflect-Explain (MORE) thinking frame. Journal of Chemical Education , 83 (4), 622-624.

May, D.B., and Etkina, E. (2002). College physics students’ epistemological self-reflection and its relationship to conceptual learning. American Journal of Physics, 70 (12), 1249-1258.

McConnell, D.A., and van der Hoeven Kraft, K. (2011). Affective domain and student learning in the geosciences. Journal of Geoscience Education , 59 (3), 106-110.

McConnell, D.A., Jones, M.H., Budd, D.A., Bykerk-Kauffman, A., Gilbert, L.A., Knight, C., Kraft, K.J., Nyman, M., Stempien, J., Vislova, T., Wirth, K.R., Perkins, D., Matheney, R.K., and Nell, R.M. (2009). Baseline data on motivation and learning strategies of students in physical geology courses at multiple institutions: GARNET Part 1, Overview. Geological Society of America Abstracts with Programs , 41 (7), 603.

McConnell, D.A., Stempien, J.A., Perkins, D., van der Hoeven Kraft, K.J., Vislova, T., Wirth, K.R., Budd, D.A., Bykerk-Kauffman, A., Gilbert, L.A., and Matheney, R.K. (2010). The little engine that could—less prior knowledge but high self-efficacy is equivalent to greater prior knowledge and low self-efficacy. Geological Society of America Abstracts with Programs , 42 (5), 191.

McCrindle, A.R., and Christensen, C.A. (1995). The impact of learning journals on metacognitive and cognitive processes and learning performance. Learning and Instruction , 5 (2), 167-185.

McKenzie, G.D., Utgard, R.O., and Lisowski, M. (1986). The importance of field trips: A geological example. Journal of College Science Teaching , 15 , 17-20.

Mestre, J. (2005). Transfer of learning from a modern multidisciplinary perspective . Greenwich, CT: Information Age.

Middlecamp, C. (2008). Chemistry in context: Goals , evidence , gaps . Paper presented at the National Research Council’s Workshop on Linking Evidence to Promising Practices in STEM Undergraduate Education, Washington, DC. Available: http://www7.nationalacademies.org/bose/PP_Middlecamp_WhitePaper.pdf .

Milliken, K.L., Barufaldi, J.P., McBride, E.F., and Choh, S.J. (2003). Design and assessment of an interactive digital tutorial for undergraduate-level sandstone petrology. Journal of Geoscience Education , 51 (4), 381-386.

Nagda, B.A., Gregerman, S.R., Jonides, J., Von Hippel, W., and Lerner, J.S. (1998). Undergraduate student-faculty research partnerships affect student retention. Review of Higher Education , 22 (1), 55-72.

Nakhleh, M.B. (1993), Are our students conceptual thinkers or algorithmic problem solvers? Journal of Chemical Education, 71 , 52-55.

National Research Council. (2007). Taking science to school: Learning and teaching science in grades K-8. Committee on Science Learning, Kindergarten Through Eighth Grade, R.A. Duschl, H. A. Schweingruber, and A.W. Shouse (Eds.). Board on Science Education, Center for Education. Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.

National Research Council. (2012). A framework for K-12 science education practices , crosscutting concepts , and core ideas . Committee on a Conceptual Framework for New K-12 Science Education Standards, Board on Science Education. Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.

Novak, J.D., Gowin, D.B., and Kahle, J.B. (1984). Learning how to learn . New York: Cambridge University Press.

Novick, L.R. (1988). Analogical transfer, problem similarity, and expertise. Journal of Experimental Psychology: Learning, Memory, and Cognition, 14 , 510-520.

Novick, L.R., and Holyoak, K.J. (1991). Mathematical problem solving by analogy. Journal of Experimental Psychology: Learning, Memory, and Cognition , 17 , 398-415.

Nurrenbern, S.C., and Pickering, M. (1987). Concept-learning versus problem-solving: Is there a difference. Journal of Chemical Education , 64 (6), 508-510.

Perkins, K.K., Adams, W.K., Pollock, S.J., Finkelstein, N.D., and Wieman, C.E. (2005). Correlating student beliefs with student learning using the Colorado Learning Attitudes about Science Survey. AIP Conference Proceedings, 790, 61-64. Pessoa, L. (2008). On the relationship between emotion and cognition. Nature Reviews Neuroscience , 9 (2), 148-158.

Petroski, H. (1996). Invention by design: How engineers get from thought to thing . Cambridge, MA: Harvard University Press.

Piaget, J. (1978). The development of thought . Oxford, UK: Basil Blackwell.

Pickering, M. (1990). Further studies on concept learning versus problem solving. Journal of Chemical Education , 67 (3), 254-255.

Pintrich, P.R., and De Groot, E.V. (1990). Motivational and self-regulated learning components of classroom academic performance. Journal of Educational Psychology , 82 (1), 33-40.

Redish, E.F., Steinberg, R.N., and Saul, J.M. (1997). The distribution and change of student expectations in introductory physics. AIP Conference Proceedings, 399, 689-698.

Reed, S.K. (1987). A structure-mapping model for word problems. Journal of Experimental Psychology: Learning, Memory, and Cognition, 13 , 124-139.

Resnick, L., and Klopfer, L. (1989). Toward the thinking curriculum: Current cognitive research . Alexandria, VA: Association for Supervision and Curriculum Development.

Rogoff, B., and Wertsch, J.V. (1984). Children’s learning in the zone of proximal development. San Francisco, CA: Jossey-Bass.

Rosengrant, D., Etkina, E., and Van Heuvelen, A. (2006). An overview of recent research on multiple representations. AIP Conference Proceedings , 883 (1), 149-152.

Ross, B.H. (1984). Remindings and their effects in learning a cognitive skill. Cognitive Psychology , 16 (3), 371-416.

Rueckert, L. (2002). Report from CUR 2002: Assessment of research. Council for Undergraduate Research Quarterly, 22, 10-11.

Russell, S.H., Hancock, M.P., and McCullough, J. (2007). Benefits of undergraduate research experiences. Science , 316 (5824), 548-549.

Salter, C.L. (2001). No bad landscape. Geographical Review , 91 , 105-112.

Sandi-Urena, S., Cooper, M.M., and Stevens, R.H. (2012). Effect of cooperative problem-based lab instruction on metacognition and problem-solving skills. Journal of Chemical Education, 89 (6), 700-706.

Schwartz, D.L., Bransford, J.D., and Sears, D. (2005). Efficiency and innovation in transfer. In J. Mestre (Ed.), Transfer of learning: Research and perspectives (pp. 1-52). Greenwich, CT: Information Age.

Semken, S. (2005). Sense of place and place-based introductory geoscience teaching for American Indian and Alaska native undergraduates. Journal of Geoscience Education , 52 (2), 149-157.

Semsar, K., Knight, J.K., Birol, G., and Smith, M.K. (2011). The Colorado Learning Attitudes about Science Survey (CLASS) for use in biology. CBE-Life Sciences Education , 10 (3), 268-278.

Sere, M.G., Journeaux, R., and Larcher, C. (1993). Learning the statistical analysis of measurement errors. International Journal of Science Education , 15 (4), 427-438.

Sibley, D.F. (2009). A cognitive framework for reasoning with scientific models. Journal of Geoscience Education , 57 (4), 255-263.

Silverthorn, D.U. (2006). Teaching and learning in the interactive classroom. American Journal of Physiology—Advances in Physiology Education , 30 (4), 135-140.

Simpson, R.D., Koballa, T.R., Oliver, J.S., and Crawley, F.E. (1994). Research on the affective dimensions of science learning. In D. Gabel (Ed.), Handbook of research on science teaching and learning (pp. 211-234). New York: Macmillan.

Slater, S.J. (2010). The educational function of an astronomy REU program as described by participating women. Ph.D. dissertation, University of Arizona, UMI ProQuest 3412598.

Slater, S.J., Slater, T.F., and Shaner, A. (2008). Impact of backwards faded scaffolding in an astronomy course for pre-service elementary teachers based on inquiry. Journal of Geoscience Education, 56 (5), 408-416.

Snow, R.E. (1989). Toward assessment of cognitive and conative structures in learning. Educational Researcher , 18 (9), 8-14.

Snow, R.E., Corno, L. and Jackson, D., III. (1996). Individual differences in affective and conative functions. In D.C. Berliner and R. Calfee (Eds.), Handbook of educational psychology (pp. 243-310). New York: Macmillan.

Sorby, S.A., and Baartmans, B.J. (2000). The development and assessment of a course for enhancing the 3-D spatial visualization skills of first-year engineering students. Journal of Engineering Education , 89 , 301-308.

Sousa, D.A. (2011). How the ELL brain learns . Thousand Oaks, CA: Corwin Press.

Stains, M., and Talanquer, V. (2008). Classification of chemical reactions: Stages of expertise. Journal of Research in Science Teaching , 45 (7), 771-793.

Storbeck, J., and Clore, G.L. (2007). On the interdependence of cognition and emotion. Cognition and Emotion , 21 (6), 1212-1237.

Sullivan, W.M. (2005). Work and integrity: The crisis and promise of professionalism in America . San Francisco: Jossey-Bass.

Svinicki, M.D. (2004). Learning and motivation in the postsecondary classroom . Bolton, MA: Anker.

Svinicki, M.D. (2011). Synthesis of the research on teaching and learning in engineering since the implementation of ABET Engineering Criteria 2000 . Paper presented at the Second Committee Meeting on the Status, Contributions, and Future Directions of Discipline-Based Education Research. Available: http://www7.nationalacademies.org/bose/DBER_Svinicki_October_Paper.pdf .

Tal, R.T. (2001). Incorporating field trips as science learning environment enrichment—an interpretive study. Learning Environments Research , 4 , 25-49.

Teichert, M.A., Tien, L.T., Anthony, S., and Rickey, D. (2008). Effects of context on students’ molecular-level ideas. International Journal of Science Education , 30 , 1095-1114.

Tien, L.T., Rickey, D., and Stacy, A.M. (1999). The M.O.R.E. thinking frame: Guiding students’ thinking in the laboratory. Journal of College Science Teaching , 28(5), 318-324.

Tien, L.T., Teichert, M.A., and Rickey, D. (2007). Effectiveness of a MORE laboratory module in prompting students to revise their molecular-level ideas about solutions. Journal of Chemical Education, 84 (1), 175-181.

Tobias, S., and Everson, H.T. (1996). Assessing metacognitive knowledge monitoring . College Board Report No. 96-1. New York: College Board.

Trigwell, K., Prosser, M., Marton, F., and Runesson, U. (2001). Views of learning, teaching practices, and conceptions of problem solving. In N. Hativa and P. Goodyear (Eds.), Teacher thinking , beliefs and knowledge in higher education (pp. 241-254). Dordrecht, The Netherlands: Kluwer Academic.

van der Hoeven Kraft, K.J., Srogi, L., Husman, J., Semken, S., and Fuhrman, M. (2011). Engaging students to learn through the affective domain: A new framework for teaching in the geosciences. Journal of Geoscience Education , 59 (2), 71-84.

Van Heuvelen, A., and Zou, X. (2001). Multiple representations of work and energy processes, American Journal of Physics , 69 (2), 184.

Vislova, T., Mcconnell, D., Stempien, J.A., Benson, W.M., Matheney, R.K., van Der Hoeven Kraft, K.J., Budd, D., Gilbert, L.A., Bykerk-Kauffman, A., and Wirth, K.R. (2010). Role of gender in student affect in introductory physical geology courses at multiple institutions. Geological Society of America Abstracts with Programs , 42 (5).

Vygotsky, L.S. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press.

Weinstein, C.E., Husman, J., and Dierking, D.R. (2000). Self-regulation interventions with a focus on learning strategies. In B. Monique, R.P. Paul, M. Zeidner, M. Boekaerts, and P.R. Pintrich (Eds.), Handbook of self-regulation (pp. 727-747). San Diego, CA: Academic Press.

Yerushalmi, E., Henderson, C., Heller, K., Heller, P., and Kuo, V. (2007). Physics faculty beliefs and values about the teaching and learning of problem solving. I. Mapping the common core. Physical Review Special Topics—Physics Education Research , 3 (2), 020109-1-020109-31.

Zuckerman, H. (1992). Scientific elite: Nobel laureates’ mutual influences (2nd ed.). New York: Pergamon Press.

Austin, A.E. (1994). Understanding and assessing faculty cultures and climates. New Directions for Institutional Research , 84 , 47-63.

Austin, A.E. (1996). Institutional and departmental cultures: The relationship between teaching and research. New Directions for Institutional Research , 90 , 57-66.

Austin, A.E. (2011). Promoting evidence-based change in undergraduate science education . Paper presented at the Fourth Committee Meeting on Status, Contributions, and Future Directions of Discipline-Based Education Research. Available: http://www7.nationalacademies.org/bose/DBER_Austin_March_Paper.pdf .

Austin, A.E., Connolly, M., and Colbeck, C.L. (2008). Strategies for preparing integrated faculty: The Center for the Integration of Research, Teaching, and Learning. In C.L. Colbeck, K.A. O’Meara, and A.E. (Eds.), Educating integrated professionals: Theory and practice on preparation for the professoriate : New directions for teaching and learning (pp. 69-81). San Francisco, CA: Jossey-Bass.

Beichner, R.J. (2008). The SCALE-UP project: A student-centered active learning environment for undergraduate programs . Paper presented at the National Research Council’s Workshop on Linking Evidence to Promising Practices in STEM Undergraduate Education, Washington, DC. Available: http://www7.nationalacademies.org/bose/Beichner_CommissionedPaper.pdf .

Bess, J. (1978). Socialization of graduate students. Research in Higher Education, 8 , 289-317.

Blackburn. R.T., and Lawrence, J.H. (1995). Faculty at work: Motivation, expectation, satisfaction . Baltimore, MD: Johns Hopkins University Press.

Borrego, M., Froyd, J.E., and Hall, T.S. (2010). Diffusion of engineering education innovations: A survey of awareness and adoption rates in U.S. engineering departments. Journal of Engineering Education , 99 (3), 185-207.

Braxton, J., Lucky, W., and Holland, P. (2002). Institutionalizing a broader view of scholarship through Boyer’s four domains. In ASHE-ERIC Higher Education Report (vol. 29, no. 2). San Francisco, CA: Jossey-Bass/John Wiley & Sons.

Bronfenbrenner, V. (1979). The ecology of human development: Experiments by nature and design. Cambridge, MA: Harvard University Press.

Brower, A., and Inkelas, K. (2010). Living-learning programs: One high-impact educational practice we know a lot about. Liberal Education 96 .

Bunce, D.M., Havanki, K., and VandenPlas, J.R. (2008). A theory-based evaluation of POGIL workshops: Providing a clearer picture of POGIL adoption. In R.S. Moog and J.N. Spencer (Eds.), Process-oriented guided inquiry learning (POGIL) (pp. 100-113). Washington, DC: American Chemical Society.

Clark, S., and Corcoran, M. (1986). Perspectives on the professional socialization of women faculty. Journal of Higher Education, 57 , 20-43.

Cohen, D.K., and Hill, H. (2001). Learning policy: When state education reform works. New Haven, CT: Yale University Press.

Cummings, K. (2008). The Rensselaer studio model for learning and teaching: What have we learned? Paper presented at the National Research Council’s Workshop on Linking Evidence to Promising Practices in STEM Undergraduate Education, Washington, DC. Available: http://www7.nationalacademies.org/bose/Cummings_CommissionedPaper.pdf .

DeAngelo, L., Hurtado, S., Pryor, J., Kelly, K., Santos, J., and Korn, W. (2009). The American college teacher: National norms for the 2007-2008 HERI faculty survey. Los Angeles, CA: Higher Education Research Institute, UCLA.

DeNeef, A.L. (2002). The Preparing Future Faculty Program: What difference does it make? Washington, DC: Association of American Colleges and Universities.

Ebert-May, D., Derting, T.L., Hodder, J., Momsen, J.L., Long, T.M., and Jardeleza, S.E. (2011). What we say is not what we do: Effective evaluation of faculty professional development programs. Bioscience , 61 (7), 550-558.

Eckel, P., and Kezar, A. (2003). Taking the reins: Institutional transformation in higher education . Phoenix, AZ: ACE-ORYX Press.

Fairweather, J. (2005). Beyond the rhetoric: Trends in the relative value of teaching and research in faculty salaries. Journal of Higher Education , 76 , 401-422.

Fairweather, J. (2008, October). Linking evidence and promising practices in science , technology , engineering , and mathematics (STEM) undergraduate education: A status report for the National Academies National Research Council Board on Science Education. Paper presented at the National Research Council’s Workshop on Linking Evidence to Promising Practices in STEM Undergraduate Education, Washington, DC. Available: http://www7.nationalacademies.org/bose/Fairweather_CommissionedPaper.pdf .

Fairweather, J., and Paulson, K. (1996). Industrial experience: Its role in faculty commitment to teaching. Journal of Engineering Education, 85 , 209-216.

Fairweather, J., and Paulson, K. (2005). The evolution of scientific fields in American universities: Disciplinary differences, institutional isomorphism . Paper presented at the Annual Conference of the Consortium of Higher Education Researchers, Jyvaskyla, Finland.

Felder, R.M., and Brent, R. (2010). The National Effective Teaching Institute: Assessment of impact and implications for faculty development. Journal of Engineering Education , 99 (2), 121-134.

Feuer, M.J., Towne, L., and Shavelson, R.J. (2002). Scientific culture and educational research. Educational Researcher, 31, 4-14.

Fisher, P.D., Fairweather, J.S., and Amey, M.J. (2003). Systemic reform in undergraduate engineering education: The role of collective responsibility. International Journal of Engineering Education , 19 (6), 768-776.

Garet, M.S., Porter, A.C., Desimone, L., Birman, B.F. and Yoon K.S. (2001). What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 38 , 915-945.

Gess-Newsome, J., Southerland, S.A., Johnston, A., and Woodbury, S. (2003). Educational reform, personal practical theories, and dissatisfaction: The anatomy of change in college science teaching. American Educational Research Journal , 40 (3), 731-767.

Gibbs, G., and Coffey, M. (2004). The impact of training of university teachers on their teaching skills, their approach to teaching and the approach to learning of their students. Active Learning in Higher Education , 5 (1), 87-100.

Handelsman, J., Ebert-May, D., Beichner, R., Bruns, P., Chang, A., DeHaan, R., Gentile, J., Lauffer, S., Stewart, J., Tilghman, S.M., and Wood, W.B. (2004). Education. Scientific teaching. Science , 304 (5670), 521-522.

Hativa, N. (1995). The department-wide approach to improving faculty instruction in higher education: A qualitative evaluation. Research in Higher Education , 36 (4), 377-413.

Henderson, C. (2008). Promoting instructional change in new faculty: An evaluation of the Physics and Astronomy New Faculty Workshop. American Journal of Physics , 76 (2), 179-187.

Henderson, C., and Dancy, M.H. (2009). Impact of physics education research on the teaching of introductory quantitative physics in the United States. Physical Review Special Topics—Physics Education Research , 5 (2), 020107-1–020107-9.

Henderson, C., Beach, A., and Finkelstein, N.D. (2011). Facilitating change in undergraduate STEM instructional practices: An analytic review of the literature. Journal of Research in Science Teaching, 48 (8), 952-984.

Ho, A., Watkins, D., and Kelly, M. (2001). The conceptual change approach to improving teaching and learning: An evaluation of a Hong Kong staff development programme. Higher Education , 42 (2), 143-169.

Kezar, A. (Ed.). (2009). Rethinking leadership in a complex, multicultural, and global environment: New concepts and models for higher education . Sterling, VA: Stylus.

Krane, K. (2008). The workshop for new faculty in physics and astronomy. Presented at the National Research Council’s Workshop on Linking Evidence to Promising Practices in STEM Undergraduate Education, Washington, DC. Available: http://www7.nationalacademies.org/bose/Krane_Presentation_Workshop2.pdf .

Lattuca, L., Terenzini, P., Volkwein, F. (2006). Engineering change: A study of the impact of EC2000 . Baltimore, MD: ABET.

Leslie, D.W. (2002). Resolving the dispute: Teaching is academe’s core value. Journal of Higher Education , 73 (1), 49-73.

Lohmann, J., and Froyd, F. (2010, October). Chronological and ontological development of engineering education as a field of scientific inquiry . Paper presented at the Second Meeting of the Committee on the Status, Contributions, and Future Directions of Discipline-Based Education Research, Washington, DC. Available: http://www7.nationalacademies.org/bose/DBER_Lohmann_Froyd_October_Paper.pdf .

Loucks-Horsley, S., Hewson, P.W., Love, N., and Stiles, K.E. (2009). Designing professional development for teachers of science and mathematics (3rd ed.). Thousand Oaks, CA: Corwin Press.

Macdonald, R.H., Manduca, C.A., Mogk, D.W., and Tewksbury, B.J. (2004). On the Cutting Edge: Improving learning by enhancing teaching in the geosciences. In Invention and impact: Building excellence in undergraduate science, technology, engineering, and mathematics (STEM) education . Washington, DC: American Association for the Advancement of Science.

Manduca, C.A., Mogk, D.W., and Stillings, N. (2004). Bringing research on learning to the geosciences . Report from a workshop Sponsored by the National Science Foundation and the Johnson Foundation. Northfield, MN: Carleton College, Science Education Resource Center. Available: http://serc.carleton.edu/files/research_on_learning/ROL0304_2004.pdf .

Mazur, E. (1997). Peer instruction: A user’s manual . Upper Saddle River, NJ: Prentice Hall.

McLaughlin, J., Iverson, E., Kirkendall, R., Bruckner, M., and Manduca, C.A. (2010). Evaluation report of On the Cutting Edge . Available: http://serc.carleton.edu/files/NAGTWorkshops/2009_cutting_edge_evaluation_1265409435.pdf .

National Research Council. (2007). Taking science to school: Learning and teaching science in grades K-8. Committee on a Conceptual Framework for New K-12 Science Education Standard, Board on Science Education. Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.

National Research Council. (2009). A new biology for the 21st century. Committee on a New Biology for the 21st Century: Ensuring the United States Leads the Coming Biology Revolution. Board on Life Sciences, Division on Earth and Life Studies. Washington, DC: The National Academies Press.

National Science Board. (1986). Undergraduate science , mathematics and engineering education. NSB 86010. Arlington, VA: National Science Foundation.

Pfund, C., Miller, S., Brenner, K., Bruns, P., Chang, A., Ebert-May, D., Fagen, A.P., Gentile, J., Gossens, S., Khan, I.M., Labov, J.B., Pribbenow, C.M., Susman, M., Tong, L., Wright, R., Yuan, R.T., Wood, W.B., and Handelsman, J. (2009). Professional development. Summer institute to improve university science teaching. Science , 324 (5926), 470-471.

Quinn-Patton, M. (2010). Developmental evaluation: Applying complexity concepts to enhance innovation and use . New York: Guilford Press.

Rogers, E.M. (2003). Diffusion of innovations (5th ed.). New York: The Free Press.

Schuster, J.H., and Finkelstein, M.J. (2006). The American faculty: The restructuring of academic work and careers . Baltimore, MD: Johns Hopkins University Press.

Seymour, E. (2001). Tracking the process of change in U.S. undergraduate education in science, mathematics, engineering, and technology. Science Education, 86 , 79-105.

Smith, B., MacGregor, J., Matthews, R., and Gabelnick, F. (2004). Learning communities: Reforming undergraduate education . San Francisco, CA: Jossey-Bass.

Tobias, S. (1992). Revitalizing undergraduate science: Why some things work and most don’t. An occasional paper on neglected problems in science education. Tucson, AZ: Research Corporation.

U.S. Department of Education. (2005). 2004 National Survey of Postsecondary Faculty (NSOPF:04) report on faculty and instructional staff in fall 2003 . Washington, DC: National Center for Education Statistics.

Wieman, C., Perkins, K., and Gilbert, S. (2010). Transforming science education at large research universities: A case study in progress. Change: The Magazine of Higher Learning , 42 (2), 7-14.

Wilson, S.M. (2011, May). Effective STEM teacher preparation, instruction, and professional development. Paper presented at the workshop of the National Research Council’s Committee on Highly Successful Schools or Programs for K-12 STEM Education, Washington, DC. Available: http://www7.nationalacademies.org/bose/STEM_Schools_Wilson_Paper_May2011.pdf .

Wlodkowski, R.J. (1999). Enhancing adult motivation to learn: A guide to improving instruction and increasing learner achievement (revised edition). San Francisco, CA: Jossey-Bass.

Wood, W.B., and Gentile, J.M. (2003a). Teaching in a research context. Science , 302 (5650), 1510.

Wood, W.B., and Gentile, J.M. (2003b). Meeting report: The first National Academies Summer Institute for Undergraduate Education in Biology. [Congresses]. Cell Biology Education, 2 (4), 207-209.

Yerushalmi, E., Cohen, E., Heller, K., Heller, P., and Henderson, C. (2010). Instructors’ reasons for choosing problem features in a calculus-based introductory physics course. Physical Review Special Topics—Physics Education Research , 6 (2), 020108-1–020108-11.

This page intentionally left blank.

The National Science Foundation funded a synthesis study on the status, contributions, and future direction of discipline-based education research (DBER) in physics, biological sciences, geosciences, and chemistry. DBER combines knowledge of teaching and learning with deep knowledge of discipline-specific science content. It describes the discipline-specific difficulties learners face and the specialized intellectual and instructional resources that can facilitate student understanding.

Discipline-Based Education Research is based on a 30-month study built on two workshops held in 2008 to explore evidence on promising practices in undergraduate science, technology, engineering, and mathematics (STEM) education. This book asks questions that are essential to advancing DBER and broadening its impact on undergraduate science teaching and learning. The book provides empirical research on undergraduate teaching and learning in the sciences, explores the extent to which this research currently influences undergraduate instruction, and identifies the intellectual and material resources required to further develop DBER.

Discipline-Based Education Research provides guidance for future DBER research. In addition, the findings and recommendations of this report may invite, if not assist, post-secondary institutions to increase interest and research activity in DBER and improve its quality and usefulness across all natural science disciples, as well as guide instruction and assessment across natural science courses to improve student learning. The book brings greater focus to issues of student attrition in the natural sciences that are related to the quality of instruction. Discipline-Based Education Research will be of interest to educators, policy makers, researchers, scholars, decision makers in universities, government agencies, curriculum developers, research sponsors, and education advocacy groups.

READ FREE ONLINE

Welcome to OpenBook!

You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

Do you want to take a quick tour of the OpenBook's features?

Show this book's table of contents , where you can jump to any chapter by name.

...or use these buttons to go back to the previous chapter or skip to the next one.

Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

Switch between the Original Pages , where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

To search the entire text of this book, type in your search term here and press Enter .

Share a link to this book page on your preferred social network or via email.

View our suggested citation for this chapter.

Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

Get Email Updates

Do you enjoy reading reports from the Academies online for free ? Sign up for email notifications and we'll let you know about new publications in your areas of interest when they're released.

• Open access
• Published: 12 July 2024

Evaluating the perceived impact and legacy of master’s degree level research in the allied health professions: a UK-wide cross-sectional survey

• Terry Cordrey 1 , 2 ,
• Amanda Thomas 3 ,
• Elizabeth King 1 , 2 &
• Owen Gustafson 1 , 2  

BMC Medical Education volume  24 , Article number:  750 ( 2024 ) Cite this article

Metrics details

Post graduate master’s degree qualifications are increasingly required to advance allied health profession careers in education, clinical practice, leadership, and research. Successful awards are dependent on completion of a research dissertation project. Despite the high volume of experience gained and research undertaken at this level, the benefits and impact are not well understood. Our study aimed to evaluate the perceived impact and legacy of master’s degree training and research on allied health profession practice and research activity.

A cross-sectional online survey design was used to collect data from allied health professionals working in the United Kingdom who had completed a postgraduate master’s degree. Participants were recruited voluntarily using social media and clinical interest group advertisement. Data was collected between October and December 2022 and was analysed using descriptive statistics and narrative content analysis. Informed consent was gained, and the study was approved by the university research ethics committee.

Eighty-four responses were received from nine allied health professions with paramedics and physiotherapists forming the majority (57%) of respondents. Primary motivation for completion of the master’s degree was for clinical career progression ( n  = 44, 52.4%) and formation of the research dissertation question was predominantly sourced from individual ideas ( n  = 58, 69%). Formal research output was low with 27.4% ( n  = 23) of projects published in peer reviewed journal and a third of projects reporting no output or dissemination at all. Perceived impact was rated highest in individual learning outcomes, such as improving confidence and capability in clinical practice and research skills. Ongoing research engagement and activity was high with over two thirds ( n  = 57, 67.9%) involved in formal research projects.

The focus of master's degree level research was largely self-generated with the highest perceived impact on individual outcomes rather than broader clinical service and organisation influence. Formal output from master’s research was low, but ongoing research engagement and activity was high suggesting master’s degree training is an under-recognised source for AHP research capacity building. Future research should investigate the potential benefits of better coordinated and prioritised research at master’s degree level on professional and organisational impact.

Peer Review reports

Higher levels of research engagement by healthcare organisations and clinicians are associated with improved organisational performance and clinical outcomes [ 1 , 2 , 3 ]. The Allied Health Professions (AHPs) comprise one third of the health and social care workforce in the United Kingdom and when engaged in research, offer substantial benefit to population health and organisational performance [ 4 ]. The strategic focus on AHP research has grown substantially in recent years. This includes the first ever national research and innovation strategy for AHPs in England, as well as clear strategic intention through AHP clinical research networks hosted by the National Institute for Health Research [ 5 , 6 ]. These strategies reinforce the need for capacity building, engagement, and cultural improvements for advancing AHP research. Realising these ambitions has, to date, been limited by insufficient funding, career infrastructure, and organisational support [ 7 ].

Alongside the strategic ambitions for AHP research, is the increasing requirement for post-graduate master’s degree qualifications for career progression in academic, leadership and clinically advanced AHP roles. For example, 69% of Advanced Clinical Practitioners (ACP’s) state the requirement for master’s degree qualification for their current ACP role [ 8 ].

With few exceptions, a master’s degree award is dependent on the successful completion of a supervised research dissertation project. This is usually accompanied by taught research methodology to support the development of research knowledge and skills. Master’s degree research ideas are conceived in a variety of ways, either as stand-alone projects, supplied by a university academic as one part of a larger programme of work, or developed in collaboration with a health service [ 9 ]. AHP research projects developed collaboratively between health and academic centres are more likely to be widely disseminated, impactful on clinical practice, and lead to further research compared to projects undertaken exclusively within a university setting [ 10 ].

Despite the high cumulation of training and research at this level over the years, the broader impact on clinical services, employing organisations, and the wider research community is currently unknown [ 11 ]. Beyond the fulfilment of individual learning objectives, it is difficult to determine what real-world impact AHP master’s research offers in terms of original knowledge contribution. Similarly, the rate of conversion of AHP master’s degree research to peer reviewed publications or conference proceedings remains unexplored [ 12 ]. This situation risks a low return on investment in terms of the generation and translation of knowledge to address the challenges faced by AHPs in healthcare practice [ 13 ]. Responsible practice in AHP post graduate training and research should, in part, be concerned with reducing waste that arises from decisions about what research to prioritise, as well as educational benefit to the individuals [ 14 ]. Aligning and coordinating more AHP master's degree research activity through collaboration may prevent AHP dissertations entering the “relevance waste quadrant” [ 15 ]. Models of portfolio research, which are coordinated efforts to address the highest priority knowledge gaps through research collaborations, represent an alternative approach to the current system [ 16 ].

The primary aim of our study is to evaluate the perceived impact and sustained effect of master’s degree research dissertation projects on AHP research capacity, capability, and clinical practice. In doing this, we have set out five supporting objectives:

To understand how master’s degree research dissertation questions were determined.

To establish the rate of conversion of master’s degree research to traditional measures of research output and dissemination.

To establish whether successful completion of master’s degree research promotes the maturation of ongoing research active clinicians.

To determine the perceived impact of research skills developed through master’s degree completion on AHP research capacity building within individuals and organisations.

To determine the perceived impact of master’s degree research on clinical practice and services.

An online cross-sectional survey design was chosen as the method to conduct this study, and it is being reported according to the consensus-based checklist for reporting of survey studies [ 17 ]. A bespoke survey was constructed using Microsoft Forms software and was hosted online via Microsoft Office 365. The survey comprised 27 questions arranged into sections to collect data on participant demographics, and the experience, outcomes, and perceived impact of master’s degree training and completion of a research dissertation project (see additional file 2 in supplementary information). To develop the survey, a pilot survey was undertaken using four qualified AHP volunteers to appraise the structure, content, and readability of the questions. Feedback from the pilot was used to revise and finalise the survey.

The target population were AHPs, which is an umbrella term for fourteen different professions usually employed in a variety of roles across health, care, academic, and voluntary sectors ( https://www.england.nhs.uk/ahp/role/ ). Participants were eligible to take part if they were 1) qualified AHPs currently working in the United Kingdom (UK), 2) held a post graduate master’s degree award, and 3) were able to provide informed consent. Participants were ineligible if their master’s degree was obtained as a pre-registration qualification, and they did not meet the other inclusion criteria. A target sample size of 139 was calculated by estimating the proportion of all registered AHPs in the UK holding a master’s degree qualification. This estimation was determined by profiling the qualifications of AHP staff in two large National Health Service (NHS) teaching hospitals. To account for a sampling calculation error, a confidence interval (95%) and margin of error (5%) threshold were applied accordingly (see additional file 3 in supplementary information).

Participant recruitment was achieved through advertising on social media platforms, and via newsletters and bulletins circulated by AHP professional and clinical interest groups. Participant information was provided outlining the study details, anonymity of survey responses, and the requirement to provide informed consent and eligibility at the start of the online survey. Those taking part were asked to reflect on their experiences of completing a post graduate master’s degree and research dissertation project in relation to its impact and legacy. The ‘one response per participant’ feature was enabled to prevented multiple completion of the survey by the same participant. The survey was live for data collection for three months running from October to the end of December 2022. During data collection, several efforts were made to promote the survey through social media to increase participation.

The survey data was analysed in two ways. First, descriptive statistics were used to analyse numerical, multiple choice, and ordinal scale data. Second, free text responses providing reflective accounts and experiences underwent coding and content analysis using NVivo software (version 12).

This study was approved by the university research ethics committee (registration number: 221613) and was conducted in accordance with the principles of good clinical practice.

The survey received 84 responses from nine of the fourteen allied health professions, which represents 60% of the target sample of 139. The majority of responses were from physiotherapists ( n  = 40, 47.6%) and respondents had been qualified for a median (IQR) of 18 years (12–23). Respondents worked in a variety of clinical specialties, with emergency/pre-hospital medicine ( n  = 18, 21.4%), neurology ( n  = 12, 14.3%) and critical care ( n  = 11, 13.1%) the most common. Most respondents had completed their master’s degree after 2010 ( n  = 68, 81%) and were employed at band 6 grade when starting ( n  = 39, 46.4%). Most respondents worked in the NHS ( n  = 78, 92.3%) and had undertaken a Master of Science (MSc) award ( n  = 70, 83.3%). Most participants were employed in a higher paid position after completing their master's degree ( n  = 62, 73.8%). The full characteristics of the respondents are detailed in Table  1 .

Respondents predominantly formed their dissertation research questions from their own area of interest (Table  2 ). Less than 10% of the dissertation questions were based on published research priorities or set by the Higher Education Institute (HEI), regional or local healthcare organisation/collaborative ( n  = 7, 8.3%). A variety of methodologies were used to conduct the master’s research dissertation with evidence synthesis being the most common ( n  = 30, 35.7%).

Formal research output from the dissertations was low (Table  2 ). Half the dissertations were presented at a local research symposium ( n  = 44, 52.4%), 27.4% ( n  = 23) were published in a peer reviewed journal, and over a third of dissertations had no output at all ( n  = 30, 35.7%). Master’s degree programmes contributing to the peer reviewed publications as a proportion of students were Master by Research (MRes) ( n  = 5, 45.5%), and MSc ( n  = 18, 25.7%).

Of the dissertations formed through the individual's own ideas, 27.6% ( n  = 16) were published in a peer reviewed journal, compared to 57.1% ( n  = 4) of those set through research priorities, or the HEI/healthcare organisation. The most common methodologies published in a peer review journal were evidence synthesis ( n  = 7, 30.4%), qualitative interviews/focus groups ( n  = 6, 26.1%) and quantitative experimental studies ( n  = 6, 26.1%). The methodology of dissertation projects with the highest proportion of peer reviewed journal publication was qualitative interviews/focus groups ( n  = 7, 36.8%).

The respondents reported their master's degree dissertation as having a positive impact on their professional development (Fig.  1 ). Qualitative content analysis of the free text responses demonstrated that respondents felt the dissertation increased their research capability and confidence at multiple stages of the research process while providing opportunities for networking and collaborations.

Perceived impact of master’s degree research on professional and clinical service development

Most participants continued to engage in research activities after their dissertation ( n  = 65, 77.4%) through supporting others ( n  = 63, 75%), taking part in formal research projects ( n  = 57, 67.9%) and publishing research papers ( n  = 41, 48.8%) (Table  3 ). Less than ten percent (9.5%, n  = 8) reported being deterred from undertaking further research (Fig.  1 ).

The wider perceived impact of the dissertation on services in which the respondents worked was more varied (Fig.  1 ). Improved service user outcome/experience and team practice was reported by 60.7% ( n  = 51) and 53.6% ( n  = 45) respectively. Analysis of free text responses demonstrated wide ranging perceived impact on services from no local impact to improved team education, service delivery and application of evidence-based practice.

Our study evaluated the perceived impact of master's degree level research on AHP professional development, research capacity, and clinical practice. Our findings indicate a relatively low level of dissemination and formal output arising from master’s degree research, but a high perception of impact on individual AHPs and the clinical services in which they work. The level of ongoing engagement in research activity following master’s degree completion was high indicating a positive legacy in this respect. The degree to which this meaningfully contributes to AHP research capacity building requires further investigation.

The majority (69%) of master’s degree research questions were developed from the respondent’s own ideas rather than drawing on published research priorities or collaborations between health and academic organisations. The limited use of research priorities may be explained by a potential lack of awareness. A qualitative study of 95 AHPs working in Australia found that in the absence of a recognised framework to guide research prioritisation, individual clinicians conducted research in areas important to them [ 18 ]. Pursuing individual preferences in this way stemmed from evaluations of their personal work, departmental policies or procedures, models of care innovation, and a clear preference for research which “tested solutions”. Similarly, Amalkumaran et al. (2016) explored critical care research priorities and found that research topics suggested by professional sub-groups tended to be related to their daily practice rather than broader research priorities [ 19 ].

It is also possible that the choice of research question is influenced by the career motivation of the individual AHP. A UK wide cross-sectional survey of AHPs working in health and social care reported primary motivators for research participation were to develop skills (80%) and increase job satisfaction (63%), rather than contribute to the prioritised evidential knowledge base [ 20 ]. Davis et al. (2019) also recognise this self-actualising motivation for research participation in their AHP cohort [ 18 ]. It is possible that the debut, non-commissioned research activity introduced by master’s level academic programmes emphasises process over content , decreasing the alignment of research activity with known research priorities.

We found a low conversion rate from master’s dissertation completion to formal research output. This is well illustrated in that just one in four (27.4%) master’s theses resulted in a peer-reviewed publication. Similar publication rates have been reported in master’s students of other healthcare disciplines; these are also considered low by way of expected research output [ 21 ]. Understanding this further is challenging due to the limited research in this subject area, which suggests a lack of interest and/or perceived importance. However, there are two key issues that arguably counter this view. First, master’s degree research projects are typically approved by a university research ethics committee, and thus are guided by the principle that the value in their conduct and knowledge contribution should outweigh the burden or risks to participants [ 22 ]. Fidelity to this principle can only be meaningfully appraised if the results are published for wider critical evaluation. Second, AHP skill and success level in research activities, such as writing for peer-reviewed publication is widely and consistently reported as low [ 23 , 24 , 25 ]. This clearly represents an area for improvement for AHPs and failing to challenge the development of this skill in those undertaking post graduate level research seems counter intuitive. Higher rates of master’s degree research publication could offer a meaningful contribution to AHP research capacity building, since our findings suggest there is continued engagement in research activity from this group beyond completion of their studies.

Respondents to our survey indicated a good level of research engagement after master’s degree training. Over three quarters reported continued involvement in research beyond the completion of their programme. This finding supports the idea that research education is a key lever and greatly needed to successfully build AHP research capacity [ 26 , 27 ]. However, the degree to which master’s degree training translates to growth in the research capacity of individuals has not been subject to causal investigation. Proxy indicators of individual research capacity from our cohort can be found in the self-reported high levels of research confidence and capability derived from master’s degree training (Fig.  1 ) and ongoing research activity. This activity included 60% taking part in formal research projects, around half had published research papers, and over a third had embarked on a higher research degree. The lack of previous research in this area makes it challenging to fully contextualise our findings, but in conducting our study, we have set out a benchmark for the perceived impact of masters degree training on individual AHP research capacity for future investigation.

We explored higher level outcomes of master’s degree training on research capacity building, such as those that might influence policy, career pathways, and organisational practice. Using the Kirkpatrick-Barr model of educational outcomes, we found the activity and outcomes from our cohort aligned best to an individual learner level [ 28 ]. This finding is typical of outcomes from education at this level, which centre largely on the individuals through self-reported satisfaction and perceptions of learning [ 29 ]. Understanding the impact of research education and training in relation to higher Kirkpatrick-Barr outcomes requires objective and longitudinal evaluation of research metrics and impact at organisational and system level [ 30 ]. This is likely to include contributions to larger programmes of work requiring large grant awards, significant publications, and translation of those research findings to health organisation and system level innovation [ 31 , 32 ]. Research capacity building at this level is known to be challenging due to the inherent complexities involving political, financial, structural, and cultural factors [ 33 ]. To overcome this, the use of theoretical frameworks has been suggested to help conceptualise and integrate a culture and proliferation of AHP research at various health system structural levels [ 34 ]. The positioning of AHP master’s degree training and research activity as part of this may foster greater academic-health system collaboration for professional, service user, and population benefit [ 35 ].

The perceived impact of master’s degree research included improvements to service user outcomes, clinical pathways, and organisational policies and/or guidelines. Research impact, defined as the demonstrable benefit of research to individuals and society, is complex and requires wide stakeholder engagement to determine whether research has addressed known priorities through effective translation of knowledge from its findings [ 36 ]. The self-generated research questions and low level of dissemination and output reported by our cohort suggests a degree of dissonance between the level of perceived impact versus what is measurably impactful to clinical services and end users. This difference may be explained by the challenges in defining and quantifying research impact for novice researchers, which is described as an ambiguous and subjective concept [ 37 ]. It is therefore not surprising to see the highest levels of reported impact from our cohort was on their own professional development in terms of improved confidence, leadership and research capability, and clinical practice development. Without a more objective assessment of the wider impact from the research undertaken at this level, it is difficult to reconcile its actual impact. The emergence of assessment frameworks, such as the visible impact of research tool, make it accessible for relatively inexperienced researchers to understand how their research has led to visible changes and impact on services and other research consumers [ 38 ].

Strengths and limitations

A key strength of our study lies in its novelty; we believe it to be the first to evaluate the perceived impact of research undertaken by AHPs at master’s degree level. This represents an important first step in highlighting the conduct and contribution of research undertaken at this level, as well providing opportunities to improve future practice and impact. There are several limitations to our study. We only managed to recruit 60% of our target sample via a non-probability sampling technique, which included a lack of representation from five of the 14 professions. This means our findings are vulnerable to sampling bias by potentially excluding AHPs who do not use social media or subscribe to clinical interest groups, which were the two main platforms for our recruitment. Our recruitment practice and the method of a self-reporting survey means our findings are not generalisable to the wider AHP population and they should be interpreted with these limitations in mind. A further limitation is the disproportionate representation of two of the fourteen allied health professions. Responses from paramedics and physiotherapists constituted 57% of our data with very few responses from seven other professions and no responses from five of the professions.

The perceived impact of AHP master’s degree training and research was highest on individual development rather than service and organisation outcomes. This is likely to derive from the individual motivation in undertaking post-graduate study and self-determined research dissertation focus. Whilst the formal research output arising from the master’s research was relatively low, the legacy in terms of ongoing research engagement and activity was positive indicating that master’s degree completion maybe an under-recognised source of AHP research capacity building. Our study provides novel insights into the perceived impact of AHP master’s degree level research. Future research should explore the feasibility and benefits of coordinating AHP master’s degree research with local or national priorities to understand the impact beyond that realised at an individual level.

Availability of data and materials

All data generated or analysed during this study are included in this published article [and its supplementary information files].

Abbreviations

Allied Health Professions

Advanced Clinical Practitioner

United Kingdom

National Health Service

Qualitative data analysis software

Interquartile Range

Master of Science

Higher Education Institute

Masters by Research

Master of Art

Doctor of Philosophy

Boaz A, Hanney S, Jones T, et al. Does the engagement of clinicians and organisations in research improve healthcare performance: a three-stage review. BMJ Open. 2015;5:e009415. https://doi.org/10.1136/bmjopen-2015-009415 .

Article   Google Scholar  

Hanney S, Boaz A, Jones T, Soper B. Engagement in research: an innovative three-stage review of the benefits for health-care performance. Southampton: NIHR Journals Library; 2013.

Google Scholar  

Harding K, Lynch L, Porter J, Taylor NF. Organisational benefits of a strong research culture in a health service: a systematic review. Aust Health Rev. 2017;41(1):45–53. https://doi.org/10.1071/AH15180 .

Harris J, Grafton K, Cooke J. Developing a consolidated research framework for clinical allied health professionals practising in the UK. BMC Health Serv Res. 2020;20:852. https://doi.org/10.1186/s12913-020-05650-3 .

Health Education England. Allied health professions research and innovation strategy. NHS England; 2022.  https://www.hee.nhs.uk/our-work/allied-health-professions/enable-workforce/allied-health-professions%E2%80%99-research-innovation-strategy-england . Accessed 21 Jan 2023.

National Institute of Health Research. NIHR Clinical Research Network (CRN) allied health professionals strategy 2018–2020. 2018. https://www.nihr.ac.uk/documents/nihr-crn-allied-health-professionals-strategy-2018-2020/11530 . Accessed 4 Jan 2023.

Pager S, Holden L, Golenko X. Motivators, enablers, and barriers to building allied health research capacity. J Multidiscip Healthc. 2012;5:53–9. https://doi.org/10.2147/JMDH.S27638 .

Stewart-Lord A, Beanlands C, Khine R, Shamah S, Sinclair N, Woods S, Woznitza N, Baillie L. The role and development of advanced clinical practice within allied health professions: a mixed method study. J Multidiscip Healthc. 2020;25(13):1705–15. https://doi.org/10.2147/JMDH.S267083 .

Angus RL, Hattingh HL, Weir KA. Experiences of hospital allied health professionals in collaborative student research projects: a qualitative study. BMC Health Serv Res. 2022;22:729. https://doi.org/10.1186/s12913-022-08119-7 .

Whelan K, Thomas JE, Madden AM. Student research projects: the experiences of student dietitians, university faculty members, and collaborators. J Am Diet Assoc. 2007;107(9):1567–74. https://doi.org/10.1016/j.jada.2007.06.009 .

Zwanikken PA, Dieleman M, Samaranayake D, et al. A systematic review of outcome and impact of Master’s in health and health care. BMC Med Educ. 2013;13:18. https://doi.org/10.1186/1472-6920-13-18 .

Murray C, Judd D, Snyder P. Evaluation of a post-professional master’s program in allied health. J Allied Health. 2001;30(4):223–8.

Salbach NM, O’Brien K, Evans C, Yoshida K. Dissemination of student research in a Canadian master of science in physical therapy programme. Physiother Can. 2013;65(2):154–7. https://doi.org/10.3138/ptc.2012-18 .

Choi T, Palermo C, Sarkar M, Whitton J, Rees C, Clemans A. Priority setting in higher education research using a mixed methods approach. Higher Ed Res Dev. 2022;42(4):816–30. https://doi.org/10.1080/07294360.2022.2082389 .

Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gülmezoglu AM, Howells DW, Ioannidis JP, Oliver S. How to increase value and reduce waste when research priorities are set. Lancet. 2014;383(9912):156–65. https://doi.org/10.1016/S0140-6736(13)62229-1 .

Nyström ME, Karltun J, Keller C, et al. Collaborative and partnership research for improvement of health and social services: researcher’s experiences from 20 projects. Health Res Policy Sys. 2018;16:46. https://doi.org/10.1186/s12961-018-0322-0 .

Sharma A, Minh Duc N, Luu Lam Thang T, et al. A consensus-based checklist for reporting of survey studies (CROSS). J Gen Intern Med. 2021;36:3179–87. https://doi.org/10.1007/s11606-021-06737-1 .

Davis A, Lee D, Wenzel L, Haines T. Setting research priorities within allied health: what do clinicians think? Int J Allied Health Sci Pract. 2019;17(1):Article 6.

Arulkumaran N, Reay H, Brett SJ, JLA Intensive Care Research Priority Setting Partnership. Research priorities by professional background - a detailed analysis of the James Lind Alliance Priority Setting Partnership. J Intensive Care Soc. 2016;17(2):111–6. https://doi.org/10.1177/1751143715609954 .

Comer C, Collings R, McCracken A, Payne C, Moore A. Allied health professionals’ perceptions of research in the United Kingdom national health service: a survey of research capacity and culture. BMC Health Serv Res. 2022;22(1):1094. https://doi.org/10.1186/s12913-022-08465-6 .

Resta RG, McCarthy Veach P, Charles S, Vogel K, Blase T, Palmer CG. Publishing a master’s thesis: a guide for novice authors. J Genet Couns. 2010;19(3):217–27. https://doi.org/10.1007/s10897-009-9276-2 .

United Kingdom Research and Innovation. Framework for research ethics. 2021. https://www.ukri.org/councils/esrc/guidance-for-applicants/research-ethics-guidance/framework-for-research-ethics/our-core-principles/ . Accessed 23 Jan 2023.

Cordrey T, King E, Pilkington E, et al. Exploring research capacity and culture of allied health professionals: a mixed methods evaluation. BMC Health Serv Res. 2022;22:85. https://doi.org/10.1186/s12913-022-07480-x .

Frakking T, Craswell A, Clayton A, Waugh J. Evaluation of research capacity and culture of health professionals working with women, children and families at an Australian Public Hospital: a cross sectional observational study. J Multidiscip Healthc. 2021;14:2755–66. https://doi.org/10.2147/JMDH.S330647 . Published 2021 Oct 1.

Matus J, Wenke R, Hughes I, Mickan S. Evaluation of the research capacity and culture of allied health professionals in a large regional public health service. J Multidiscip Healthc. 2019;12(83):96. https://doi.org/10.2147/JMDH.S178696 .

King O, West E, Lee S, Glenister K, Quilliam C, Wong Shee A, Beks H. Research education and training for nurses and allied health professionals: a systematic scoping review. BMC Med Educ. 2022;22(1):385. https://doi.org/10.1186/s12909-022-03406-7 .

King E, Cordrey T, Gustafson O. Exploring individual character traits and behaviours of clinical academic allied health professionals: a qualitative study. BMC Health Serv Res. 2023;23:1025. https://doi.org/10.1186/s12913-023-10044-2 .

Barr H, Koppel I, Reeves S, Hammick M, Freeth DS. Effective interprofessional education: argument, assumption and evidence (promoting partnership for health). Chichester: John Wiley & Sons, Incorporated; 2005.

Book   Google Scholar  

Famure O, Batoy B, Minkovich M, Liyanage I, Kim SJ. Evaluation of a professional development course on research methods for healthcare professionals. Healthc Manage Forum. 2021;34(3):186–92.

Madi M, Hamzeh H, Griffiths M, et al. Exploring taught masters education for healthcare practitioners: a systematic review of literature. BMC Med Educ. 2019;19:340. https://doi.org/10.1186/s12909-019-1768-7 .

Duncanson K, Webster EL, Schmidt DD. Impact of a remotely delivered, writing for publication program on publication outcomes of novice researchers. Rural Remote Health. 2018;18(2):4468.

Schmidt D, Duncanson K, Webster E, Saurman E, Lyle D. Critical realist exploration of long-term outcomes, impacts and skill development from an Australian rural research capacity building programme: a qualitative study. BMJ Open. 2022;12(12):e065972.

Whitehouse CL, Copping J, Morris P, Guledew D, Chilson B, Gray R, et al. An organisational approach to building research capacity among nurses, midwives and allied health professionals (NMAHPs) in clinical practice. Int Pract Dev J. 2022;12(2):1.

Slade SC, Philip K, Morris ME. Frameworks for embedding a research culture in allied health practice: a rapid review. Health Res Policy Sys. 2018;16:29. https://doi.org/10.1186/s12961-018-0304-2 .

Matus J, Walker A, Mickan S. Research capacity building frameworks for allied health professionals - a systematic review. BMC Health Serv Res. 2018;18(1):716. https://doi.org/10.1186/s12913-018-3518-7 .

Reed MS, Ferré M, Martin-Ortega J, Blanche R, Lawford-Rolfe R, Dallimer M, Holden J. Evaluating impact from research: a methodological framework. Res Policy. 2021;50(4):104147. https://doi.org/10.1016/j.respol.2020.104147

Belcher B, Halliwell J. Conceptualizing the elements of research impact: towards semantic standards. Humanit Soc Sci Commun. 2021;8:183. https://doi.org/10.1057/s41599-021-00854-2 .

Jones NL, Cooke J, Holliday J. Making occupational therapy research visible: amplifying and elevating the contribution and impacts. Br J Occup Ther. 2021;84(4):197–9. https://doi.org/10.1177/0308022620988473 .

Download references

Acknowledgements

The authors would like to thank all the allied health professionals who gave their time to participate in this survey.

Dr Owen Gustafson, Clinical Doctoral Research Fellow (NIHR301569), is funded by Health Education England (HEE)/National Institute for Health Research (NIHR). The views expressed in this publication are those of the authors and not necessarily those of the NIHR, NHS or the UK Department of Health and Social Care.

Author information

Authors and affiliations.

Oxford Allied Health Professions Research and Innovation Unit, Oxford University Hospitals NHS Foundation Trust, Oxford, OX3 9DU, UK

Terry Cordrey, Elizabeth King & Owen Gustafson

Centre for Movement, Occupational, and Rehabilitation Sciences, Oxford Brookes University, Oxford, OX3 0BP, UK

Barts Health NHS Trust, Royal London Hospital, London, E1 1BB, UK

Amanda Thomas

You can also search for this author in PubMed   Google Scholar

Contributions

All authors conceived and designed the study. All authors designed the survey content and structure. TC prepared the online survey. All authors promoted recruitment to the survey. EK and OG undertook data analysis and interpretation. TC prepared Fig.  1 . OG prepared Tables 1 ,  2 and  3 . AT wrote the background and part of the discussion. TC wrote the abstract, methods, part of the discussion, and conclusion. EK and OG wrote the results. All authors reviewed the manuscript and consented to publication.

Corresponding author

Correspondence to Terry Cordrey .

Ethics declarations

Ethics approval and consent to participate.

The study was approved by Oxford Brookes University research ethics committee and assigned registration number: 221613. This study was conducted according to the relevant guidelines and regulations of the Declaration of Helsinki. Survey respondents were required to read the participant information sheet and provide informed consent prior to taking part.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., supplementary material 3., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Cordrey, T., Thomas, A., King, E. et al. Evaluating the perceived impact and legacy of master’s degree level research in the allied health professions: a UK-wide cross-sectional survey. BMC Med Educ 24 , 750 (2024). https://doi.org/10.1186/s12909-024-05582-0

Download citation

Received : 20 March 2023

Accepted : 21 May 2024

Published : 12 July 2024

DOI : https://doi.org/10.1186/s12909-024-05582-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Master’s degree
• Research training
• Allied health professions
• Research capacity
• Research impact

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]

• Skip to main content
• Skip to main navigation

Alternative Assessments

What and why of alternative assessment.

Are the dozens of research papers all starting to blur together? Are the scantron bubbles beginning to haunt your dreams? More than likely, they are for students too. In moving away from traditional forms of assessment it is becoming more common practice, and highly desired, by students, teachers, and the professional world to extend the life of assessments past a single moment. Alternative Assessment may offer new ways for you and your students to explore subject matter in unique, and holistically beneficial ways.

Although carrying its own importance and necessity in achieving specific outcomes,  summative assessments  do have distinct drawbacks (Williams, 2014). These typically include:

• Tedious completion and grading for professors and students
• Narrow learning outcomes
• A focus on the grade, rather than the process (for more on this see  Ungrading )
• Disposable products that are never seen by student or teacher again
• Instances of concern for academic integrity

Alternative assessment offers solutions to these drawbacks and speak to emerging needs of college graduates. The professional world seeks college graduates who possess not only discipline-specific factual knowledge but also the problem solving, collaborative, and interdisciplinary skills that cannot be achieved by artificial intelligence advances (Binkley et al., 2012). Considering this, mixed method approaches to assessment are becoming necessary in the college environment (Hains-Wesson et al., 2020). Incorporating aspects of summative ,  formative , and alternative assessment can help to expand and enhance learning outcomes for the students as well as provide new experiences for the professor.

Types of Alternative Assessment

Peer/self-assessment.

In this form, often written products are exchanged among peers for assessment. While the professor may provide the rubric, training, and criteria for the consistency of peer assessment, the “assessment” of the assignment is conducted by fellow students. This offers opportunity for the product to be reviewed by multiple people before any final submission to the professor and allows students to view their peers work. This exposure among peers can help to facilitate new connections and perspectives that students may be able to communicate among themselves in a way that had been missed in the course prior. Self-assessment can be constructed in the same way (with a rubric, criteria, and revisions) but offers students an opportunity to reflect on their own thought process and externally process. Additionally, pairing peer/self-assessment may allow students to review their peer’s work and then reflect upon their own in a new light (Wen & Tsai, 2006). These activities can also facilitate  community within the classroom .

Authentic Assessment

This form of assessment aims to create “authentic” experiences that require practical, context-driven approaches with assessment as learning opportunities (Gulikers et al., 2004). When moving away from quantitative means of assessment, it can be difficult to concretely define authentic assessment in practice. Guliker et al. provides a five-dimensional theoretical framework:

• “An authentic task is a problem task that confronts students with activities that are also carried out in professional practice.”
• Physical Context: “ Where we are, often if not always, determines how we do something, and often the real place is dirtier (literally and figuratively) than safe learning environments… Authentic assessment often deals with high fidelity contexts.”
• Social Context:  “In real life, working together is often the rule rather than the exception… learning and performing out of school mostly takes place in a social system.”
• Quality product students would be asked to produce in real life
• Demonstration that permits making valid inferences about the underlying competencies
• Full array of tasks and multiple indicators of learning
• Presentation of work either written or orally to other people”
• Criteria and standards: “ Setting criteria and making them explicit and transparent to learners beforehand is important in authentic assessment, because this guides learning… and employees usually know on what criteria their performances will be judged.”

Work Integrated Learning (WIL)

This form of assessment aims to not only  imitate  authentic field experiences, but actually participates in these experiences outside of the classroom. As such, WIL is technically a form of authentic assessment – but takes it even further than in class experiences. Although common place in many vocationally oriented programs (such as social work internships or education student teaching), WIL is not limited to these types of curricula. Implementing WIL experiences into a course (or program) entails distinct challenges—specifically ensuring rigor and a means of using effective assessment practices (Ajjawi et al., 2020). As institutions are ultimately responsible for the WIL assessments reflecting intentional learning outcomes, it is important to put great care into ensuring the alignment of WIL assessments.

Multi-media

Technology has become a larger part of the college experience, not only in the classroom, but in the way that course assignments are completed. These various forms of media provide additional resources and creativity for alternative assessment. Developing electronic portfolios, creating video essays/reflections, music videos, or other digital products that require student creativity and engaging with course material is an effective means of using technology to aid in learning outcomes. The language of the learning outcomes do not necessarily have to change, but the criteria can. If large multimedia projects seem overwhelming or unachievable at a scale for your course, offering these as options for extra credit or alternatives to existing assignments is a good trial run for their implementation.

Crafting your own Alternative Assessment

Some of these suggestions may already be in practice or more inherent to your discipline. For others, jumping into these may seem daunting. It is not necessary to attempt to overhaul a course overnight. From these categories, there are endless possibilities for how to incorporate aspects into a single class session, or into a semester long capstone project. To hear from your fellow faculty more in depth on ways they are incorporating Alternative Assessment into their classrooms, view this  Seminar for Excellence in Teaching .

• What can I do next class?  Have students write a self-reflection on that day’s topic on how it relates to their career path, what aspects of the topic they feel they have grasped well, or what areas they feel like they still do not understand well.
• What can I do next unit?  Craft the end of unit assessment to be something other than an exam or essay. Try to holistically assess student learning as they are experiencing it (e.g., have students put together a portfolio—in a journal or online—that has them describe/narrate as they view their complete understanding of that unit.)
• What can I do next semester?  Design each end of unit assessment as a segment that culminates into an end of course capstone project. The final product could combine audio, visual, and/or physical components that reflects products created within this discipline.
• What can I do across semesters?  Create a project or experience that builds on itself as more semesters participate in the activity—such that data collected, products created, or ideas generated create a continuous “living” assessment (e.g., literature reviews/meta-analyses that continually build based on new publications and expanding datasets to monitor trends through time).

Below are key characteristics and examples of alternative assessments compared with traditional approaches. While this is not exhaustive, it provides an insight to the ways in which you can begin to transform your own course assignments and assessments.

Figure 1: Traditional and Alternative Assessment Characteristics (Rojas Serrano, 2017)

Challenges and Champions of Alternative Assessment

In a similar frame of mind as formative assessment ,  alternative assessment seeks to provide a means of assessing student learning in real time, rather than as a snapshot such as in summative assessment. Alternative assessment is less focused on grades and more focused on student process and thinking, allowing instructors to more clearly into the minds of their students and their learning. Other objectives you may be aiming for (including  Universal Design for Learning  and  compassionate teaching ) can be incorporated, if not enhanced, by employing alternative assessment. This has distinct advantages and challenges in the way it is realized in the classroom (Stasio et al., 2019). While these are important considerations, they are meant to provide context and mindful considerations as you explore these alternatives.

Challenges:

• Ensuring academic rigor
• Restructuring understanding of student’s role in learning
• Requires time for development
• Trial and error may be necessary
• Aligning course outcomes with assessment tasks
• Source of motivation for students connecting with their areas of study
• Holistic approaches to subject matter
• Applied skills and products for student portfolios
• Build collaboration and opportunities for living course work
• Places learning outcomes in the foreground

Ajjawi, R., Tai, J., Huu Nghia, T. le, Boud, D., Johnson, L., & Patrick, C. J. (2020). Aligning assessment with the needs of work-integrated learning: the challenges of authentic assessment in a complex context.  Assessment and Evaluation in Higher Education ,  45 (2), 304–316. https://doi.org/10.1080/02602938.2019.1639613

Binkley, M., Erstad, O., Herman, J., Raizen, S., Ripley, M., Miller-Ricci, M., & Rumble, M. (2012). Defining Twenty-First Century Skills. In  Assessment and Teaching of 21st Century Skills  (pp. 17–66). Springer Netherlands. https://doi.org/10.1007/978-94-007-2324-5_2

Gulikers, J. T. M., Bastiaens, T. J., & Kirschner, P. A. (2004). A Five-Dimensional Framework for Authentic Assessment.  ETR&D ,  52 (3), 67–86.

Hains-Wesson, R., Pollard, V., Kaider, F., & Young, K. (2020). STEM academic teachers’ experiences of undertaking authentic assessment-led reform: a mixed method approach.  Studies in Higher Education ,  45 (9), 1797–1808. https://doi.org/10.1080/03075079.2019.1593350

Rojas Serrano, J. (2017). Making sense of alternative assessment in a qualitative evaluation system.  Profile Issues in Teachers’ Professional Development ,  19 (2), 73–85. https://doi.org/10.15446/profile.v19n2.57178

Stasio, M. di, Ranieri, M., & Bruni, I. (2019). Assessing is not a joke. Alternative assessment practices in higher education.  Form@re - Open Journal per La Formazione in Rete ,  19 (3), 106–118. https://doi.org/http://dx.doi.org/10.13128/form-7488

Wen, M. L., & Tsai, C.-C. (2006). University students’ perceptions of and attitudes toward (online) peer assessment.  Higher Education ,  51 , 27–44. https://doi.org/DOI 10.1007/s10734-004-6375-8

Williams, P. (2014). Teaching in Higher Education Squaring the circle: a new alternative to alternative-assessment.  Teaching in Higher Education , 565–577. https://doi.org/10.1080/13562517.2014.882894

Academy for Teaching and Learning

Moody Library, Suite 201

One Bear Place Box 97189 Waco, TX 76798-7189

• General Information
• Academics & Research
• Administration
• Gateways for ...
• About Baylor
• Give to Baylor
• Pro Futuris
• Social Media
• College of Arts & Sciences
• Diana R. Garland School of Social Work
• George W. Truett Theological Seminary
• Graduate School
• Hankamer School of Business
• Honors College
• Louise Herrington School of Nursing
• Research at Baylor University
• Robbins College of Health and Human Sciences
• School of Education
• School of Engineering & Computer Science
• School of Music
• University Libraries, Museums, and the Press
• More Academics
• Compliance, Risk and Safety
• Human Resources
• Marketing and Communications
• Office of General Counsel
• Office of the President
• Office of the Provost
• Operations, Finance & Administration
• Senior Administration
• Student Life
• University Advancement
• Undergraduate Admissions
• Graduate Admissions
• Baylor Law School Admissions
• Social Work Graduate Programs
• George W. Truett Theological Seminary Admissions
• Online Graduate Professional Education
• Virtual Tour
• Visit Campus
• Alumni & Friends
• Faculty & Staff
• Prospective Faculty & Staff
• Prospective Students
• Anonymous Reporting
• Annual Fire Safety and Security Notice
• Cost of Attendance
• Digital Privacy
• Legal Disclosures
• Mental Health Resources
• Web Accessibility

Your browser is not supported

Sorry but it looks as if your browser is out of date. To get the best experience using our site we recommend that you upgrade or switch browsers.

Find a solution

• Skip to main content
• Skip to navigation

• Back to parent navigation item
• Collections
• Sustainability in chemistry
• Simple rules
• Teacher well-being hub
• Women in chemistry
• Global science
• Escape room activities
• Decolonising chemistry teaching
• Teaching science skills
• Get the print issue
• RSC Education

• More navigation items

Challenge your learners to improve their NMR interpretation

• No comments

Use this vast, online spectroscopy resource to provide students with essential practice

Nuclear magnetic resonance (NMR) spectroscopy is a versatile tool used in a wide range of scientific disciplines, including chemistry, materials science, biology and medicine.

Source: Composite image, all © Shutterstock

Share NMR Challenge with your learners to provide more than 500 real NMR spectra and their solutions

Beyond structural elucidation, scientists use NMR spectroscopy to study reaction kinetics, reaction mechanisms and intermolecular interactions. Interpreting NMR spectra is a core skill and consequently an integral component of teaching post- 16 spectroscopy .

Students first encounter NMR spectra in secondary school, where they use it to determine the structures of simple organic molecules as part of problem-solving style questions. As an information-rich technique, students can use multiple components of spectra to derive structural information, such as the number, intensity and shape of signals, the chemical shift and J-coupling constants. While often challenging, these questions can develop critical thinking skills .

In response to the limited number of educational resources available to practise NMR interpretation, researchers developed  NMR Challenge , an online resource containing more than 500 real NMR spectra of 200 organic compounds. Users can submit solutions using a structure drawing tool and get confirmation of the correct solution.

In response to the limited number of educational resources available to practise NMR interpretation, researchers developed NMR Challenge (bit.ly/3XPsAJd), an online resource containing more than 500 real NMR spectra of 200 organic compounds. Users can submit solutions using a structure drawing tool and get confirmation of the correct solution.

They identified many common mistakes, possibly reflecting students’ misunderstandings of the basic principles of NMR spectra

The creators of the site organised the problems by complexity. The basic level contains 148 1D spectra ( 1 H, 13 C and 19 F) and are appropriate for secondary school students. The advanced level contains 2D spectra. Within the basic and advanced levels, there is further classification: easy, moderate and hard.

In a new paper , the researchers – who developed the site – analysed the success rates of all 200 tasks on NMR Challenge, consisting of more than 428,000 solutions from around the globe, the largest database of responses to NMR assignments. Through this analysis, they identified many common mistakes, possibly reflecting students’ misunderstandings of the basic principles of NMR spectra. The following three case studies summarise the main findings.

Interpreting the findings

Case study 1 : when considering isomeric esters, the researchers found students to be competent in using spectral information to recognise molecular fragments, such as a monosubstituted benzene ring, an ethyl chain and the ester group. However, students were less able to use the chemical shift values to identify the connectivity of these fragments.

Case study 2 : when considering substitution patterns in disubstituted benzenes, students only recognised para-substituted structures well. The more complex splitting pattern in the 1 H spectrum for ortho- and meta-disubstituted benzenes is the likely cause. However, you can use the number of signals on 13 C spectra to distinguish homodisubstituted benzenes, which learners rarely considered.

Case study 3 : many spectra also included intramolecular interactions, for example the formation of a hydrogen bond between an exchangeable hydrogen atom (for example, OH) and a hydrogen bond acceptor (for example, O in carbonyl). Students often misidentified the increase in chemical shift in the exchangeable hydrogen as they principally used the 1 H NMR spectra for identification and therefore did not draw the correct structures.

Teaching tips

Use NMR Challenge  for free in your teaching and suggest it as independent practice for your students. The resource provides feedback on submission and informs the user if their answer is correct.

The study provides further analysis in the supporting information , including the most common mistakes. You can adapt this into an additional classroom activity based on simulated-peer assessment .

The researchers identify several key observations to consider when teaching NMR spectroscopy:

• Focus on tasks that consider isomeric structures, determining their structural fragments and connections.
• Compare spectra of esters with spectra of their alcohol and carboxylic acid starting materials.
• While students are able to recognise para-disubstituted benzene rings, ortho- and meta-disubstituted are more challenging. Help students become more familiar with recognising the multiplicity of hydrogen atoms in these systems with plenty of practice.

 Z Osifová et al ,  J. Chem. Educ.,  2024, 101 , 2561–2569 ( doi.org/10.1021/acs.jchemed.4c00092 )

Fraser Scott

 Z Osifová  et al ,  J. Chem. Educ.,  2024,  101 , 2561–2569 ( doi.org/10.1021/acs.jchemed.4c00092 )

More Fraser Scott

Reduce cognitive load with an augmented reality learning environment

Understanding how students untangle intermolecular forces

Boost maths skills to improve chemistry learning

• Analytical chemistry
• Spectroscopy

Related articles

How to teach titration post-16

2024-07-08T05:32:00Z By Jo Haywood

Tips for teaching practical titration techniques and the underlying theory

How to teach chromatography at post-16

2024-03-11T04:00:00Z By Andy Markwick

Everything you need to help your students master the fundamentals of this analytical technique

Chromatography challenge | 16–18 years

By Andy Markwick

Explore analytical techniques and their applications with a chromatography investigation and research activity

No comments yet

Only registered users can comment on this article., more education research.

How best to engage students in group work

2024-06-11T05:19:00Z By David Read

Use evidence-based research to help students get the most out of group work

2024-05-16T06:00:00Z By Fraser Scott

Discover how to use augmented reality to help students visualise organic mechanisms 

How visuospatial thinking boosts chemistry understanding

2024-04-18T06:07:00Z By David Read

Encourage your students to use their hands to help them get to grips with complex chemistry concepts

• Contributors
• Print issue
• Email alerts

Site powered by Webvision Cloud