policy cycle research paper

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Springer Nature - PMC COVID-19 Collection

Policy capacities and effective policy design: a review

Ishani mukherjee.

1 School of Social Sciences, Singapore Management University, 90 Stamford Road, Level 4, Singapore, 178903 Singapore

M. Kerem Coban

2 GLODEM, Koc University, Rumelifeneri Yolu, 34450 Istanbul, Turkey

Azad Singh Bali

3 The School of Politics & International Relations; The Crawford School of Public Policy, The Australian National University, University Avenue, Acton, ACT 2600 Australia

Associated Data

Upon request.

Not applicable.

Effectiveness has been understood at three levels of analysis in the scholarly study of policy design. The first is at the systemic level indicating what entails effective formulation environments or spaces making them conducive to successful design. The second reflects more program level concerns, surrounding how policy tool portfolios or mixes can be effectively constructed to address complex policy objectives. The third is a more specific instrument level, focusing on what accounts for and constitutes the effectiveness of particular types of policy tools. Undergirding these three levels of analysis are comparative research concerns that concentrate on the capacities of government and political actors to devise and implement effective designs. This paper presents a systematic review of a largely scattered yet quickly burgeoning body of knowledge in the policy sciences, which broadly asks what capacities engender effectiveness at the multiple levels of policy design? The findings bring to light lessons about design effectiveness at the level of formulation spaces, policy mixes and policy programs. Further, this review points to a future research agenda for design studies that is sensitive to the relative orders of policy capacity, temporality and complementarities between the various dimensions of policy capacity.

Introduction: capacity considerations for effective policy design

The heart of policy design resides in the act of devising policy alternatives that meet stated government goals. While it is understood that not all policies can be carefully crafted, the policy sciences have been motivated by questions about why some policy alternatives are often developed well, while others are less so. Why do some policy choices, once formulated, effectively go forth through subsequent policymaking processes while others do not? How do some policies arise from meticulously crafted modes of formulation while others are shaped by partisan processes such as electoral or legislative bargaining (Howlett, 2011 ). Understanding factors that enable how deliberate designing of policy occurs and how superior designs can be achieved in complex issue-areas is central to the research agenda of the modern policy sciences (Howlett, 2014a , 2014b ; Howlett et al., 2017 ). The critical need to acknowledge, engage with and fully understand the capabilities underlying this exercise of good design, is also constantly escalating, especially in the face of widespread public crises.

Over the last few decades, a growing curiosity about the feasibility of formulation processes and the context within which policy choices unfold, has allowed policy scholars to gain a comparative perspective on policy design realities. Policy design is now generally defined as the purposive action of linking policy instruments with distinctly stated policy goals (Bobrow, 2006 ; Linder & Peters, 1984 ; Majone, 1975 ; May, 2003 ), stemming from the systematic endeavor to analyze how targets react or change their behaviors in response to instruments of governance. Effective design subsequently involves applying the knowledge gained about instrument-target relationships, to the creation of policies that can then predictably lead to desired policy outcomes (Bobrow & Dryzek, 1987 ; Gilabert & Lawford-Smith, 2012 ; Peters, 2018 ; Sidney, 2007 ; Weaver, 2009a , 2009b ). These activities are prefaced on the assumption that feasible polices can be realistically generated through effective design processes only when, firstly, contradictions internal to the substantive content of policy are resolved or minimized, and secondly, when the necessary capacities and capabilities to enact design procedures are in place (Bali et al., 2019 ; Mukherjee & Bali, 2019 ).

The recent scholarship in the policy sciences recognizes the first of these two emphases. For instance, studies anchored in the new design orientation explicitly focus on policy tools, how they are sequenced and assembled in mixes, how these mixes are calibrated, and their relative efficacies in meeting policy goals (del Río & Howlett, 2013 ; Howlett & Lejano, 2013 ). However, these studies have to a lesser degree raised issues about the capacity that is essential for effective policy design. In other words, experience from a variety of sectors and jurisdictions have alluded to what ‘effectiveness’ or ‘best practices’ imply for the activity of policy design, but lesser so about what capacities enable effectiveness.

Discussion of this latter topic is a largely scattered body of knowledge in the theoretical and empirical contribution of policy studies scholarship. For instance, the contemporary frameworks and theories of the policy process do not explicitly operationalize capacity as an independent variable in explaining policy outcomes (see for example Howlett et al., 2020a , 2020b for a recent review of the theories of the policy process). Here, we do not claim a ceteris paribus condition in which policy capacity is the only explanatory factor determining policy design effectiveness. While recognizing that many different determinants of policy design effectiveness exist, the article surveys the extant literature to specifically highlight the state of the knowledge on policy capacity requisites of policy design effectiveness. In doing so, the article brings to light the capacity ‘gap’ that exists in the policy design literature and draws lessons on not only what ‘effectiveness’ means at multiple levels of design but what is known to date about the capacities necessary for its enabling. The central question thus motivating this review asks what types of capacity are needed for effective policy design ? And to this aim, the article presents findings of a critical review synthesizing the existing scholarship on policy capacity and design in the policy sciences.

The article follows with an examination of the conceptual correspondence between the literatures on policy design effectiveness and policy capacity. The methodology informing this review is outlined next. In the fifth section that forms the core of our review, we consolidate the findings of our research on effective policy design spaces and instrument mixes and critically analyze these in the context of four emerging yet under-theorized themes from the scholarship on policy capacity, namely (1) the potential hierarchies in types of policy capacity, (2) the temporal dynamics within policy capacity, (3) task and agency-specific capabilities, and (4) complementarities among different types of capacities. We conclude by discussing avenues to advance a research agenda on effective design spaces and policy instrument mixes, which rigorously engages with these four themes of policy capacity.

Through this process, the paper makes two novel contributions focusing on the intersection of the policy design and policy capacity literatures. Firstly, it synthesizes the growing body of research in the policy sciences on effective policy design in terms of how particularly it discusses the necessary policy capacities that enable it. And secondly, by anchoring the review in the policy design orientation, the paper is able to identify four themes arising from the scholarly work on policy capacity that have yet to receive requisite theoretical and empirical scrutiny in the policy sciences. In doing so, we respond to repeated calls in the literature on the need to advance the scholarship and develop meaningful research questions on policy design effectiveness and the capacities that it necessitates. (Howlett & Lejano, 2013 ; Howlett et al., 2015a , 2015b ).

Understanding policy effectiveness

Policy effectiveness can be understood at three nested levels (Peters et al., 2018 ). The first relates to creating a conducive design space or an environment for policy formulation, which allows for effective policy design to occur (Howlett & Mukherjee, 2018a , 9). The second refers to developing effective policy mixes that are capable of addressing problems, and the third involves effectively designing and deploying individual policy instruments .

Effectiveness in design spaces

The essential idea is that the nature of the overall policy design space can significantly influence how effectively intended design activities occur and thus upon the likely resulting effectiveness of policy designs that emerge from them. These spaces reflect existing policy styles within a sector, are shaped by political conditions, reflect policy legacies (Howlett & Tosun, 2021 ), and therefore constrain (or enable) options available for designers. Developing policymaking spaces that are amenable to design activities involves a constant and concurrent stock-taking exercise of potential public capacities that might be pertinent in any problem-solving situation (Anderson, 1975 ). However, having an intention to be formal and analytical in designing and evaluating policy alternatives is not enough in itself to promote a design-centered process, since this also depends on the government’s ability to undertake such an analysis and to alter the status quo (Howlett & Mukherjee, 2018b ). Capacity challenges plaguing a design situation can lead to the generation of alternatives which are tenuously ‘patched’ together rather than deliberately packaged to uphold coherence and consistency (Howlett & Rayner, 2013 ).

Effectiveness in instrument mixes

While considerations for the design environment’s bearing on effective formulation have occupied the research agenda of policy tool studies in recent years, the new design orientation has contributed to a discourse on how to effectively incorporate policy mixes of policy goals and means (Briassoulis, 2005 ; Doremus, 2003 ; Gunningham et al., 1998 ; Hood, 2007 ; Howlett, 2011 ; Jordan et al., 2011 , 2012 ; Peters et al., 2005 , 2018 ; Yi & Feiock, 2012 ).

Selecting and deploying multiple instruments in the context of dedicated policy mixes ‘are all about constrained efforts to match goals and expectations both within and across categories of policy elements’ (Howlett, 2009a , 74). Achieving effectiveness with respect to deploying such mixes or policy portfolios relies on ensuring that mechanisms, calibrations, objectives and settings display ‘coherence’, ‘consistency’ and ‘congruence’ with each other (Howlett & Rayner, 2007 ). Scholars steeped in the new design orientation who are concerned with effectiveness have cautioned about how some policy mixes that are not designed in a planned fashion, can be plagued by internal inconsistencies, whereas others can be more successful in creating an internally supportive combination (del Río, 2010 ; Grabosky, 1994 ; Gunningham et al., 1998 ; Howlett & Rayner, 2007 ). This depends on how well they are able to adapt and support changing policy circumstances, as Thelen ( 2004 ) noted how the organization of macro-institutions has usually not resulted through calculated planning but rather has emerged out of processes of incremental adjustments such as ‘layering’ or ‘drift’ (Sewerin et al., 2020 ).

Effectiveness at the instrument level

While most of the research in the contemporary policy sciences have focused on issues around design spaces and instrument mixes, these has been limited, if any, comparative research on the efficacy of individual instruments and how they are calibrated (Capano and Howlett, 2020 ). At the most granular level, this third level of effectiveness focusses on the efficacy of individual policy tools and how these individual instruments are calibrated. Within this, we also need to differentiate between substantive instruments such as taxes, licenses, and subsidies; and the more indirect procedural instruments (such as competition, network structure, and royal commissions) which include administrative processes for selecting and deploying substantive tools (Capano and Howlett, 2020 ; Howlett, 2000 ).

There are at least three factors that condition the effectiveness of individual instruments and how they are calibrated. First, the extent to which substantive policy tools is supported by their procedural counterparts. Second, the extent to which critical institutional pre-requisites that condition the performance of instruments are present in policy mixes. Third, the extent of how far particular components of instruments or their calibrations can be easily adjusted in the short run and long run. This refers to changes in the settings of instruments such as adjusting tax rates or contribution rates for a pension fund. In some cases, there are sufficient ‘degrees of freedom’ to make these changes, or for them to be auto adjusting such as cost of living stabilizers, but in many cases calibrating instruments are difficult thereby undermining the effectiveness of an instrument.

Policy capacity: a brief review

Policy capacity, defined as a set of skills, competencies, and resources across government agencies to design and pursue policy goals (Rotberg, 2014 ; Howlett, 2015 ; Tiernan & Wanna, 2006 ; Wu et al., 2010 , 2015 ), has been a central research theme in public policy in recent years (Howlett and Ramesh, 2015 ; Newman et al., 2017 ; Karo & Kattel, 2018 ; Daugbjerg et al., 2018 ; Bali & Ramesh, 2019 ). In a notable first contribution, Wu et al. ( 2015 ) offer a framework to conceptualize policy capacity at multiple levels of governance. They argue that capacity can be understood as skills and competencies existing across government agencies at three nested levels: the individual (e.g., policymakers, decision-makers), the organization (e.g., an agency or a program), and at the systemic level (e.g., the whole of government or the macro level institutional, structural contexts) (Table ​ (Table1). 1 ).

Dimensions and levels of policy capacity.

Source : Adapted from Wu et al. ( 2015 ) and Howlett and Ramesh ( 2015 )

At the level of individuals occupied with policy formulation, those striving for effective design require technical know-how to conduct practical policy analysis and disseminate knowledge, while leadership and negotiation abilities are additionally relevant for those in managerial positions. Analysts also need political savvy and acumen for incorporating and accounting for various stakeholder interests and assessing political feasibility. At the level of government organizations , information mobilization capabilities to enable timely and relevant policy analysis, administrative capital for ongoing coordination between policymaking agencies, and political backing all fundamentally build overall policy capacity. At the system level, effective policy design requires institutions for knowledge creation and utilization, alongside mechanisms to coordinate across different levels of government, and overall trust and political legitimacy (Mukherjee & Howlett, 2016 ).

Howlett and Ramesh ( 2015 ) extend Wu et al.’s ( 2015 ) work on capacity drawing on the metaphor of an ‘Achilles’ Heel.’ That is, how certain types of capacities can become critical to the sustaining policy efforts and outcomes in specific modes of governance, and how any weaknesses in these ‘critical’ capacities can undermine policy efforts (Menaheim and Stein 2013 ).

Technical knowledge, for example, is a critical capacity required for the sustainable functioning of policy systems based on market-based governance. Analytical skills at the level of individual analysts and policy workers are key, and the ‘policy analytical capacity’ (Rayner et al., 2013 ; Wellstead et al., 2011 ) of government needs to be especially high to deal with complex quantitative economic and financial issues involved in regulating and steering the sector and preventing crises (Bakır & Çoban, 2019 ; Rayner et al., 2013 ; Woo et al., 2016 ). Similarly, undertaking policy design within legal systems of governance relying heavily on high levels of managerial capacities that can deter against diminishing returns of compliance or mounting non-compliance with government directives (Coban, 2020a ; May, 2005 ). Capacities at the systemic level can be especially critical in this case as governments find it difficult to enact traditional command-and-control instruments in the absence of overall public trust.

The appeal of Wu et al.’s ( 2015 ) framework lies in its inherent simplicity. Each of the nine capabilities lend themselves to, in principle, being empirically operationalized and allows analysts to assess strengths and weaknesses of governments across different types of capabilities (e.g., Bajpai and Chong, 2019 ; Saguin et al., 2018 ). Yet such simplicity also generates concerns.

First, the contribution by Wu et al. ( 2015 ) does not lend itself to drawing causal inference or developing a theory of policy capacity. Moreover, as our review demonstrates below, the mechanisms that connect indicators with specific types of capacities are not explicitly mentioned. Secondly, the current literature seems to adopt a benevolent approach to incumbents relying on or mobilizing policy capacity. 1 That is, policy capacity could also facilitate the ‘dark side’ of policymaking (Howlett, 2020 ), by advancing policymakers’ self-interested, political and/or economic ‘rent-seeking’ objectives (see Chindarkar et al., 2017 ; Howlett and Mukherjee, 2016 ). Furthermore, it can be instrumental for developing ‘placebo policies’ as ‘agenda management safety valves’ (McConnell, 2020 , 965) or for ‘hidden agendas’ (McConnell, 2018 ) to further political goals rather than addressing the core of policy problems. These represent unchartered areas, especially if we consider the challenges generated by the rise of populism and autocratization around the world (Kelemen, 2017 ; Maerz, 2020 ; Norris & Inglehart, 2019 ).

This review relies on building and scrutinizing a database of peer-reviewed journal articles that are located at the intersection of policy capacity, policy design, and effectiveness. A keyword search based on these themes was conducted on Scopus, and Thomson Reuters’ Web of Science (WoS). Scopus and WoS are two major repositories of scientific knowledge published in various forms: conference proceedings, edited book chapters, peer-reviewed journal articles. The search protocol was conducted similarly on both databases to cross-check for any duplicate journal articles, and avoided selection bias that can result from extracting data from a single database. The search covered three collections of WoS citation indexes: Social Sciences Citation Index (SSCI), Emerging Sources Citation Index (ESCI), Arts and Humanities Citation Index (A&HCI). We explicitly included ESCI and A&HCI along with SSCI given our concerns for inclusivity.

The data collection and sample selection process had four steps. The first involved searching for, ‘policy design’, ‘capacity’, ‘effectiveness’, as keywords for the topic of an article. In this focused search, we omitted a set of alternative keywords such as ‘capability’, which are mostly used in public management scholarship. More importantly, the focused search as conducted through these keywords allowed us to capture a range of terms, such as ‘governance capacity’ and ‘administrative capacity’, in which capacity has been used in the context of policy design and/or design effectiveness. As such, it should be noted that articles that incorporated such varieties of capacity, but did not directly discuss policy design were excluded from the final database. In this light, we are aware that the search focused on a designated subset in the existing policy design literature. However, this scope allowed us to fully capture the dispersed attempts made so far to deliberately link policy capacity and design effectiveness and address our express interest in showcasing the current state of the literature that is located at the intersection of policy capacity, policy design, and design effectiveness. Additionally, the search was designed to be as inclusive as possible given the time period, disciplines, and multiple databases that it incorporates.

While acknowledging the limitations of the search logic described herein, we maintain that additional keywords would result in extra layers that dilute the task of specifically exploring the policy capacity requisites for policy design effectiveness. We also note that detecting journal articles on WoS and Scopus required us to run the search several times with various combinations of these keywords. This is because research that is positioned at the intersection of policy capacity, policy design, and effectiveness is in its adolescence. We therefore combined the results of multiple searches while removing duplicate entries. Our search covered the period between 1900 and May 17, 2020, the date we ran the search on WoS and Scopus. This time period allowed for construction of an inclusive database. This search yielded a sample of 9382 sources. The second step involved filtering our initial search for journal articles that are published in English. 2 The result of this process reduced the sample to 7441 articles. In the third step, we further refined our search by filtering the articles according to various relevant (inter)disciplinary areas: ‘political science’, ‘public administration’, ‘economics’, ‘management’, ‘international relations’, ‘sociology’, ‘social sciences interdisciplinary.’ In so doing, we included articles that are not only published in political science and public administration but also in other main social science disciplines and those that were classified in the interdisciplinary social sciences category. This choice was mainly driven by inclusivity concerns and an expectation of capturing articles that may empirically or conceptually refer to policy design, policy capacity, and/or effectiveness. The result of the second stage to limit our search to relevant fields yielded 1431 journal articles.

Following the above-mentioned steps, we read titles, abstracts, and full texts to further refine the most relevant articles. Articles that had the main keywords in the topic, but were not directly related to our research questions were omitted based on a reading of their introductory sections and research questions. We omitted articles that used different forms of capacity without an explicit interest in operationalizing capacity for design effectiveness. We also omitted articles that attempted to measure or evaluate effectiveness of an instrument or program. In this regard, as our interest in this article is to make sense of what capacity for ‘effectiveness’ means at multiple levels of design, our exclusion criteria meant that we eliminated articles which presented only nominal links between policy design and policy capacity. Consequently, the final sample included 146 articles. As for coding, the sample included articles that discuss policy design as well as effectiveness. Therefore, coding had to sort according to levels of policy design and dimensions of policy capacity. This process involved two tracks. First, we coded articles to capture dimensions of policy capacity according to parameters suggested by Wu et al. ( 2018 , 6–14). Second, reading the articles served to code an article whether it did examined design space, discussing design effectiveness of a policy instrument, policy mixes reading the articles led us to code articles whether it was about design space, discussing design effectiveness of a policy instrument, policy mixes/programs, or combinations levels of policy design.

Table ​ Table2 2 and Fig.  1 summarize the results of the coding process. Articles on design space, policy mixes and programs have the highest share among those referring to level of policy design. A main observation at the outset is that there is a significant gap in the literature on studies discussing policy design and capacity at the level of individual instruments. The review included explicitly those scholarly contributions that engage with capacity considerations. Undoubtedly, the field of environmental policy (and for that matter social policy and financial policy) is replete with the discussion of singular instrument types such as taxes, social security schemes, emissions trading schemes, among others. But this review could not identify articles that expressly deal with the question of capacity and what is needed on the part of policy designers to formulate these instruments, which is a significant void that needs to be filled in future studies. Even the studies that distill the state of knowledge on effective program design, rarely discuss individual constituent policy tools.

Levels of policy design and dimensions of policy capacity

Dimensions of policy capacity and levels of policy design

On the dimensions of policy capacity, articles address analytical and political capacity more so than operational, while there is a more equal distribution of articles referring to individual, organizational, or systemic scales. In addition, our observations point to a limited number of studies that look at both organizational and individual policy capacity, as well as both political and operational policy capacities. Finally, we note that only a few studies attempt to relate policy capacity with effective design space for global public policies, instruments, and mixes/programs (Bernstein & Cashore, 2012 ; Cashore et al., 2019 ; Dare, 2018 ; Dorsch & Flachsland, 2017 ; Jordan & Huitema, 2014 ; Stone & Ladi, 2015 ; Vince & Nursey-Bray, 2016 ).

Policy capacity requisites for effective policy design: emerging trends and existing gaps

In this section, we discuss the main findings on the link between policy design effectiveness and policy capacity as revealed through the review of the literature. While these findings are discussed at the level of effective policy design spaces and effective instrument mixes, we critically examine them through the perspective of four overarching emphases that that are developing within the scholarship on policy capacity and policy design (Bali & Ramesh, 2018 , 340–341; Howlett and Ramesh, 2015 ; Capano & Howlett, 2020 ; Howlett et al., 2015a , 2015b ). These are, namely:

• Hierarchies or ‘orders’ among specific types of capacities, which indicate what kinds of capabilities are more pre-requisite and foundational to others that are more ‘second-tier’ and aspirational.
• Temporality of policy capacity endowments, or the time needed for policy capacity investments to achieve actual effectiveness outcomes.
• The distinction between task-specific and agency-specific policy capacities and how to reconcile between them; and
• The synergies and complementarities between different policy capacities.

Capacities for effective policy design spaces

Developing effective design spaces is fundamentally about ensuring that policy tools are anticipated to fit or cohere with broad governance arrangements, while delivering a means to address certain policy goals. It is argued, for example, that several variables are critical for effectiveness within collaborative modes of governance, including reconciling with ‘prior history of conflict or cooperation, the incentives for stakeholders to participate, power and resource imbalances, leadership and institutional design’ (Ansell & Gash, 2008 , 543). Similarly, the absence of clear property rights and mechanisms to enforce contracts stymie the effectiveness of hybrid governance arrangements to design suitable public–private partnerships (PPPs) in service delivery (Virani, 2019 ).

An enabling design space that is able to support the design of its constituent policy instruments signifies an environment that is marked by high analytical, operational and political capacity (Capano, 2018 ; Chindarkar et al., 2017 ). Determining exactly what capacities are required in order to develop the political and administrative spaces needed to carry out complex policy design processes is currently a subject of much interest in the field (Considine, 2012 ). In order to address these issues, it is recognized that policy designers need to be cognizant about the internal mechanisms of their polity and constituent policy sectors which can boost or undermine their ability to think systematically about policy and develop effective policies (Braathen, 2007 ; Braathen & Croci, 2005 ; Grant, 2010 ; Skodvin et al., 2010 ).

In this vein, organizations and individual policymakers need political support from the policy design spaces or environments that they occupy. For this, they derive legitimacy and authority from system-level political capacity, which subsequently creates a favorable milieu for the application of individual and organizational political capacities during the design process (Woo et al., 2015 ; Xiarchogiannopoulou, 2015 ). Political support to policymakers and interactions between policymakers and politicians have been argued as being non-substitutable when it comes to overcoming ambiguous goals and promoting managerial effectiveness, by supplying organizations with a clear understanding of their overall mandate (Meckling & Nahm, 2018 ; Stazyk & Goerdel, 2011 ).

At more individual and organizational levels, political capacity is essential for maneuvering effectively within the constraints of the design space (Hartley et al., 2015 ) and is embodied in the levels of trust, especially political trust and legitimacy within the public sector. Individual and organizational political capacity is also necessary to garner strategic stakeholder support that is vital both before and during the design process, as well as in subsequent stages of policy implementation (Bali & Ramesh, 2019 ). For example, in the case of macro-prudential policy, a suitable financial policy mix is possible in an enabling design space that is characterized by capable, analytically skilled individual central bankers that have coalition-building skills, a government committed to evidence-informed policy, and presence of inter-organizational collaboration mechanisms at system level (Bakır & Çoban, 2019 ).

This example also highlights the importance of ‘legitimation capacity’ in effective design environments (Woo et al., 2015 ; see also Pal & Clark, 2015 ). Policymakers and organizations that are highly regarded by key societal actors and receive sustained political support are able design effective policies with more accountability (Busuioc & Lodge, 2016 ; Rimkuté, 2018 ). For example, as is visible in the case of health and safety regulation in the UK, the regulatory agency’s outreach and engagement with policy targets increases its political acumen by helping to overcoming citizens’ biases, and furthering its legitimacy by shoring up societal support for future policy design (Dunlop, 2015 ). This case also underscores the dilemma that may exist between expertise-led, technocratic, and less accountable design on the one hand; and participatory, more accountable design processes on the other, and the relative effectiveness of either situation (Montpetit, 2008 ). Yet, overall high levels of trust and political support at the system level are shown in most cases to allow the design process to be endowed with necessary information and access to critical resources at the outset (Chindarkar et al., 2017 ; Hartley et al., 2015 ).

An example of this latter context is the rise of ‘big data’ analytics that has also necessitated a parallel emphasis on big data readiness at all three levels of capacity (Clarke & Craft, 2017 ; Giest, 2017 ; Giest & Mukherjee, 2018 ; Golan et al., 2017 ). For example, policy responses to the Covid-19 pandemics in countries like Singapore have included combining mobile-phone-tower data and machine learning to develop social graphs that track propinquity to improve contact-tracing (The Economist, 2020 ; see also Woo, 2020 on the Singapore case). Big data has also been used for network analysis in policy formulation (Giest, 2017 ). But, the availability of data, network analysis and modeling necessitate complex skills such as making use of software, models to produce insights that inform policy design. Moreover, related studies have repeatedly underlined that policymakers should take into consideration behavioral dimensions of policies, which becomes more likely when organizational infrastructure allows for the participatory collection as well as engagement with behavioral data and analysis (Leong & Howlett, 2020 ; Mukherjee & Mukherjee, 2018 ).

Hierarchies within types of capacities

Studies in this review of effective design spaces implicitly operationalize specific types of capacities as a spectrum of independent variables and argue that they shape policy outcomes. While this advances our understanding of how capacity is connected with notions of effectiveness, the causal mechanisms that undergird such links have not always been made clear. This can be explained to some extent by the tendency in the literature to operationalize capacity in a straightforward, often univariate manner, while ignoring possible orders or hierarchies among specific types of capacities. In other words, policy capacity can be multi-dimensional with notable interaction between foundational, first-order and more aspirational or ‘second-order’ capacities. Lodhi ( 2018 ) and Hartley and Zhang ( 2018 ), for instance, suggest a comprehensive measurement of policy capacity. Such efforts can then allow for multiple orders of capacities to be observed while and better locate the interactions between them. A focus on how policy capacities at one level can enable, prevail over those, or constrain capacities at the other two levels are neglected factors when theorizing the link between policy capacity and policy design effectiveness.

For example, if system-level policy capacity is more crucial as it constitutes the environment in which an organization or an individual policymaker operates, can it be postulated that without the acquisition of system-level capacities, even high individual or organizational policy capacity might not be sufficient for effective policy design? More research along this vein is warranted to advance our understanding about any hierarchy or orders of policy capacity and the role they play in developing effective design spaces.

Along the same lines, most studies in this review focus on operationalizing a specific type of capacity rather than considering how combinations or interactions between different types of capacities shape policy outcomes. For instance, in a context wherein system-level policy capacities are high but individual policy capacities cannot uphold organizational capacities, one may observe sub-optimal design or even non-design. Such a case could indicate that while we may consider the presence of system-level policy capacity to be detrimental for on-the-ground mobilization of organizational and/or individual policy capacity, the reverse dynamic may also be important for effective policy design.

Further, while most studies in the review have considered political capacity to play a more critical role than operational and analytical capacities, they have stopped short of developing hypothesis or propositions to attribute plausible reasons for its significance. This, in turn, stagnates any advancement in how specific types of capacities can explain and beget design effectiveness.

Temporal dynamics of capacity

There is a gap in our understanding on the temporal dynamics and change within the policy design literature (see, e.g., Capano & Howlett, 2020 ; Bali & Ramesh, 2018 ), and this lacuna is also evident in this review of necessary capacities for effect design. Temporality in the context of capacities for effective design explores changes in specific types of capacity endowments over time, to their sustained or ultimate impacts on policy outcomes. It also includes a consideration of how investments in capacity building have a latent gestation period before which they begin to affect outcomes. None of the studies in this review explicitly dwelled on the temporal dimensions of capacities, echoing the popular refrain on the largely atheoretic discussion on policy tools and capacity (Howlett & Ramesh, 2015 ; Howlett et al., 2015a , 2015b ).

Temporality in the context of effective policy design can be conceptualized in two ways. The first is to consider the impact and scope of changes in capacity on effectiveness at different stages of design process. For example, what are the causal mechanisms by which changes in capacities contribute to changes in policy outcomes? That is, do interventions at time t 0 affect outcomes by time t n . Is the lag between t n –t 0 standardized across different types of capacities? Such lines of enquiry can inform about how individual, organizational, or system-level policy capacities change over time and result in fluctuations in the effectiveness of policy designs. For instance, the National Sample Survey Organizations of India in the 1950s was recognized globally as a center for excellence and pioneering statistical sampling techniques and methodologies, but in recent years has become mired in controversy on the quality of its statistical estimates (Banerjee et al., 2017 ).

Secondly, a discussion on temporality also implicates concerns about robustness and resilience of policy design. Robustness over time can enable policymakers, organizations or a system to endure shocks, policy surprises, and turbulence, while allowing them flexibility (Ansell et al., 2016 ; Capano & Woo, 2017 ; Howlett et al., 2018 ; Mergel et al., 2021 ). Endurance could be achieved with adaptability to structural, institutional and actor-level changes and/or evolution of existing policy capacities over time (e.g., Alaerts, 2020 ; Capano & Pavan, 2019 ; Van Der Steen et al., 2018 ). And subsequent adaptability could arise on improvements in complementarities among different types and levels of critical capacity requisites. These are particularly relevant to anticipatory policy design (Bali et al., 2019 ; Huitema et al., 2018 ; Kimbell & Vesnić-Alujević, 2020 ), especially in cases of high contextual uncertainty, as is exemplified by numerous examples of climate change impacts on agriculture or water policy domains (Nair & Howlett, 2017 ). While such a conceptualization seems plausible, the existing literature lacks a systematic understanding of what types of capacities enable design spaces to endure substantial changes in the structural and institutional contexts of policies as, for example, the Covid-19 crisis has already demonstrated (Walter, 2020 ; Weible, 2020 ).

These considerations also call for a discussion on the temporal nature of acquiring or engendering policy capacities and which of these are necessary earlier on in the design process. For example, effective policy design could be the outcome of initial improvements in individual and organizational capacities, which may later require the build-up and/or mobilization of system-level capacities. These are propositions that need to be examined to advance our understanding of whether or not individuals, organizations, or systems need to build particular capacities first for effective policy design to subsequently unfold.

Capacities for effective instrument mixes and programs

The growing intractability of contemporary challenges that governments face in areas such as health and urban planning among others has necessitated the use of multiple policy tools to be carefully and deliberately assembled in policy mixes or portfolios (Howlett & Lejano, 2013 ). This has made the task of effective policy design more challenging, as designers have to match not only policy goals and aims, but also instrument mixes and governance modes (Peters & Pierre, 2015 ; Tosun & Lang, 2017 ; Wen, 2017 ). In turn, this effort towards striving for compatibility requires a spectrum of analytical capacities that enables policymakers, organizations and political systems to employ skills pertaining to the accurate articulation of operational objectives, which in turn require an accurate interpretation of context relevant information and data. These analytical skills become fundamental to the success of sector-wide programs that may otherwise suffer from a mismatch between stated objectives and the policy tool collections that are constructed as a response. In other words, and as reported in many program-level studies, the more (or less) policymakers resemble analytically capable policy designers, the more (or less) likely they are to construct an effective mix of policies through a program. For instance , Siwale and Okoye ( 2017 ) argue that microfinance program initiatives in Zambia were ineffective largely due to limitations in policymakers’ analytical capabilities.

Besides individual and organizational policy capacities, reforms buttressed on the tenets of New Public Management (NPM) marked administrative changes in the late 1990s, which embodied a large, albeit skewed, emphasis on the kinds of capabilities that are necessary for policy success. With this transformation, policy capacity to design and steer policies became truncated, as states increasingly contracted out the delivery of public services to the private sector and civil society. This has been argued to have resulted in loss of policy capacity within government, in the reform era, in the form of declining skilled human resources which affect both organizational and system-level analytical and operational capacity within the state apparatus (Bakvis, 2002 ; Baskoy et al., 2011 ; Craft & Daku, 2017 ; Donahue et al., 2000 ; Howlett, 2000 , 2009b ; Lodge, 2013 ). Put differently, with the ‘hollowing out’ of the state, the changing role of the state as the primary actor in the design process has evolved into that of a policy navigator that steers the policy process and coordinates the interactions between non-state actors and those between the state and non-state actors (Lindquist, 1992 ). Policy capacity in this sense has been often supplemented by external expertise, knowledge, know-how supplied by variegated epistemic communities, think tanks, business, international organizations, scientists, non-governmental organizations, or civil society groups among others can supply (Haas, 1992 ; Stone, 2003 ).

With the externalization of knowledge and related capacities, many studies have alluded to greater participation being fundamental for effective program design that needs to be shaped in a way that is more notably open to stakeholder input and learning from that input (Borrás, 2011 ; Hoppe, 2011 , 2018 ; Jordan & Huitema, 2014 ; Vince & Nursey-Bray, 2016 ). The water quality program in the European Union (EU) is a case in point. Brown ( 2000 ) examines the EU’s operational and analytical capacity to design effective directives when it faces scientific uncertainty in the given policy area, and most importantly fluid number and quality of staff (see also Jensen, 2018 on policy capacity requisites for effective water policy in developing countries).

This case and others demonstrate that input from international organizations and local stakeholders generally tend to increase the supranational organization’s operational and analytical capacity. Echoing the call for greater participation, Mukherjee and Mukherjee ( 2018 ) determine citizen participation to be fundamental in co-production in rural sanitation programs in India, Bangladesh, and Indonesia. Lang ( 2014 ) studies analytical capacity in PPPs in which the private sector brings its own expertise to complement goals set out by policymakers. Similarly, Bengston et al. ( 2004 ) sheds light on participation of citizens and other stakeholders in urban policy in making formulation more effective. These studies all suggest that when policymakers have a tendency to underestimate or even ignore stakeholder participation and input, the effectiveness of policy design and implemented policies can decline considerably. While a few recent studies have now begun to look at particular types of capacities that different stakeholders, especially interest groups, can contribute (see Coban, 2020b ; Daugbjerg et al., 2018 ) they still fall short of addressing the benefits or challenges they can bring specifically to policy design effectiveness, thus calling for further research in this area.

Additionally, when non-state actors participate actively in the design process, this understandably has implications for the governance capacities that are available for effective policy formulation. Studies highlighting polycentric policy design processes have emphasized policy capabilities for enabling the coordination and collaboration of multiple actors. Political capacity to manage collaboration and coordination has also been called ‘collaborative capacity’ in some public management literature, within organizations or specific programs (Ansell & Gash, 2008 ; Braun, 2008a , 2008b ; Schout & Andrew, 2008 ; Weber et al., 2007 ).

In a multi-level design situation, such as policy programs, horizontal and vertical coordination of parties similarly demand high political capacity (Peters, 2015 ). Golan et al. ( 2017 ), for example, show that lack of effective coordination between the central authorities and the local authorities in the design of rural cash transfer programs that omit a considerable share of the target population, lead to reduced effectiveness of the program’s objectives. Similarly, Wen’s ( 2017 ) study on social policy in China indicates that when the central state does not coordinate policy design with the local authorities that lack policy capacity, policy design effectiveness faces substantial challenges at all levels.

Collaboration and coordination challenges have been significant in developing countries as well as in advanced economies. Williams and McNutt ( 2013 ), focusing on policy programs for climate change adaptation in the Canadian finance sector, assert network management capacities for aligning the targets of local and federal and provincial agencies to be built into the design of the programs and well before their implementation. Additionally, Skeete ( 2017 ) examines policy instrument mixes that regulate carbon emissions emanating from diesel use in the European Union (EU). The author finds that lack of coordination between member states and EU authorities, besides leading to inherent flexibilities of the regulatory framework, also leads to fuel taxes failing to achieve original climate policy goals. Similarly, Spendzharova ( 2016 ) maintains that disconnect between EU member states and EU authorities in the design of banking structure reforms after the global financial crisis leads to a mismatch in design processes in terms of prioritizing domestic reforms vis-à-vis EU level financial reforms.

Complementarities in policy capacities

Such studies on policy instrument mixes and programs highlight the primary role of analytical capacity in developing and deploying effective instrument mixes. However, it can be insufficient if not operating alongside suitable organizational and political capacities, which ultimately determine how successfully they are implemented (Bali et al., 2019 ; Mukherjee & Bali, 2019 ). In other words, analytical capabilities are enhanced or sharpened by operational and political capacity endowments at the level of organizations. This is not surprising as policy design is ultimately a political activity and requires individual policymakers to strategically operate within a broad community of policy stakeholders and organizations (Peters, 2015 ).

For example, Mukherjee and Giest ( 2019 ) show how individual policy entrepreneurs’ capacity to form and maintain coalitions has enabled effective use of individual, organizational and system-level capacities in digital transformation in the EU. Similarly, Ramesh and Bali ( 2019 ) demonstrate how operational capabilities in Singapore’s health system were amplified by sustained political capacity and trust in government. However, these studies and others in this review do not develop generalizable propositions that can be empirically examined on the complementarities and synergies among different types of capacities in different contexts. That is, the aggregate impact of a series of specific capacity endowments is larger than their individual impacts (Wu et al., 2015 ). Similarly, do critical deficits in capacities affect outcomes? (Howlett & Ramesh, 2015 ). These theoretical gaps are particularly visible given that developing policy designs that harness synergies and complementarities among tools is a central theme in the new design orientation (Howlett et al., 2015a , Howlett et al., 2015b ).

One way to address this missing link is to canvass the recent advances around policy success in the public management literature. For instance, design effectiveness is intrinsically related to policy success, as ‘successful policy often resides in policy design and the diligent work undertaken’ (McConnell, 2017 , 17). These themes have been interrogated further in a series of studies that aim to advance what is described as ‘positive public administration’ (Compton & ‘t Hart, 2019 ; Luetjens et al., 2019 ; Douglas et al., 2019 ), which define success across four broad dimensions: if it achieves its goals (i.e., programmatic success), produces largely supported socially appropriate outcomes (i.e., process success), contributes to problem-solving capacity and enhance legitimacy (i.e., political success), and is robust (i.e., endurance) (Ibid, 5).

Connecting groups of capacities with specific dimensions of success can allow analysts to develop proposals around complementarities in capacities to be then examined empirically. For example, policy success could be less likely when operational capacity at system level in the form of coordination mechanisms both within the state and between the state and non-state actors is not established and/or mobilized. Testable claims that emerge from this debate are that if these conditions are not met, enabling political and processual success may not emerge leading to incongruent policy goals and tools. Cumulatively, these outcomes may result in failures in programmatic and endurance terms, bringing about policy (instrument) fiascos (Bovens & ‘t Hart, 2016 ). This in turn provides a richer understanding of the types of capacities required for developing and deploying effective mixes.

Task and agency-specific capacities

There is a tendency in the literature and in contemporary debates to use ‘policy capacity’ as a catchall phrase (Wu et al., 2015 ). An avenue to overcome this simplification is to engage rigorously with the ‘capacity for what’ question (Bali & Ramesh, 2018 ). That is, to identify, ex ante, and theorize task-specific and agency-specific capacities needed for routine but complex tasks in contemporary service delivery such as contracting, managing PPPs, and administering pension funds; and accomplishing these effectively during periods of extreme uncertainties and volatility such as crises (Capano et al., 2020 ; Stirling, 2010 ).

The new design orientation has set up a tall order for effectiveness in program designs whereby designs must be coordinated, coherent, reduce contingent liabilities, and avoid Type 1&2 errors, among others (Bali & Ramesh,  2017 ; 2018 ; Howlett, 2018 ). For example, while network governance may be well suited to policy design for sensitive issues such as elderly care or parental supervision (Pestoff et al., 2012 ) in other situations, civil society may not be well enough organized or endowed in order to generate beneficial network modes of governance off-the-ground and without initial regulatory support (Tunzelmann, 2010 ). Networks, for example, ‘will fail when governments encounter capability problems at the organizational level such as a lack of societal leadership, poor associational structures and weak state steering capacities which make adoption of network governance modes problematic’ (Howlett & Ramesh, 2014 , 324).

However, in our review there is limited, if any, theoretical discussion on the types of capacities needed to achieve these outcomes. That is, the range of capacities required to accomplish tasks such as contracting, commissioning, and collaboration while all under the umbrella of network governance require a variety of distinct capabilities and skillsets (O’Flynn, 2019 ). Failing to recognize these variations and invest in task-specific capabilities has played a key role in failed social policy reforms in many developing economies (Maurya & Ramesh, 2019 ; Virani, 2019 ). Along the same lines, variations in the capacities of agencies within government to pursue such tasks must be recognized (Bardhan, 2016 ).

Conclusion: avenues to advance the research agenda on capacity and design

This paper addresses a scattered body of knowledge in the policy sciences and aims to advance our understanding of the relationship between policy capacity and effective policy design. To this end, this paper presents a review of the existing literature that studies effective policy design through the lens of policy capacity, and argues that such a perspective offers an important starting point for scrutinizing the role of complementarities among organizational, individual, and system-level analytical, operational, and political capacities, within the broader policy sciences.

Clarifying the relationship between design effectiveness and policy capacities is central to advancing the research agenda of the new design orientation in the policy sciences. The theoretical union of these two bodies of literature, at its core, is about reiterating the problem-solving approach in the policy sciences. That is, it inspires building on the research questions surrounding how specific policy interventions are devised to address specific types of problems, with notions of what is fundamentally needed to enable these designs. The most well-intentioned efforts at policy design can be constrained by the capabilities of governments, and those involved in the design process (Mukherjee & Bali, 2019 ). Forwarding such a research agenda can further refine the generalizable hypotheses to investigate and improve policy deliberations regarding effective policy formulation, which already inform the policy sciences (Howlett & Lejano, 2013 ; Howlett et al., 2015a , Howlett et al., 2015b ). To this end, this review has provided several starting points for infusing policy design research with policy capacity concerns.

Our central thesis is that the growing body of research on policy design effectiveness, which is synthesized in this paper, remains largely descriptive and tends to confound rather than clarify the relationship between policy capacity and effective policy design. Our review points to several outstanding questions that need to be highlighted: Do individual, organizational, or system-level policy capacity change over time? Does effectiveness of policy designs and success of policies vary over time with changes in policy capacity of various types ‘spilling over’ and at different levels? Thirdly, do orders of policy capacity exist? And can we distinguish between hierarchies or levels of policy capacity, which have serious implications for effective policy design and thereby policy success (or failure). Specifically, this strand of reasoning can help distil those capacities that are fundamental at the start of policy design (t 0) before successive ones are developed at subsequent stages of policy design (at t 1 and expectedly later at t n ). Is there a hierarchy among levels of policy capacity? If yes, then what is the nature of that hierarchy and are there causal inferences that can be drawn between more fundamental ‘enabling’ capacities and more aspirational ‘second-tier’ capacities? And, how does such a hierarchy impact effectiveness of policy design and determine policy success (or failure)? Finally, given the lack of focus on policy capacity requisites for effective individual policy instrument design, does, and if so, how policy capacity enable effective policy instruments?

Scholarly efforts to engage with these questions can be a generative exercise, signposting new areas for theoretical exploration and empirical testing. In this concluding section, we briefly comment on two avenues to synthesize our critique, by engaging with the two respective levels of policy effectiveness that have been explored in this paper.

Effective policy design spaces: situating capacity in theories of the policy process

A central theme in the policy design literature, which pervades all studies covered in this review, is that an enabling design space provides a platform for successful policy design, as such spaces are supported by significant capacity endowments, which not only improve policy deliberation but also allows designers to best navigate changing and often volatile design contexts (Howlett & Mukherjee, 2018a , 2018b ; Howlett et al., 2018 ; Peters et al., 2018 ; Rahman et al., 2019 ). However, most of this discussion remains largely divorced from mainstream theories and frameworks of the policy process, especially those that explain policy formulation and deliberation styles of governments. If our goal is to advance our understanding of effective design spaces, and what capacities engender them, we need to locate capacity within frameworks and theories of the policy process that are focused on them. For instance, interrogating the role of capacity in incrementalism, the policy narrative framework, or the advocacy coalition framework can generate theoretically grounded propositions and empirical testing on specific mechanisms through which capacity shapes design spaces.

Effective instrument mixes and programs: developing capacity as an independent variable

Another avenue to engage with questions relating to hierarchies, complementarities and temporal dynamics of specific types of capacities raised earlier in this paper is to explicitly canvass policy capacity as a system of independent variables, and to examine its causal impact on policy outcomes. However, as Peters ( 2020 ) states, this is challenging to do especially in the context of policy design as its impact is intermediated by many exogenous factors (Peters, 2020 ). And, as has been noted earlier, the links between specific types of capacities and how they are empirically operationalized are not always clear. Nonetheless, these methodological shortcomings can be managed to some extent by through in-depth critical case studies (see Yee & Liu, 2021 ), or focusing on comparisons among most similar cases (see Yan & Saguin, 2021 ), and avoiding sweeping comparisons that are characteristic in studies of comparative public policy. Similarly, limitations around how capacity is empirically operationalized can be managed by encouraging problem or policy-specific capacity studies. For example, Bajpai and Chong ( 2019 ) extend Wu et al.’s ( 2015 ) framework to study foreign policy capacity. Similarly, Bali and Ramesh ( 2021 ) operationalize different types of capacities to sustain health reform.

Dealing with capacity as explanatory variables would allow analysts to engage with questions around hierarchies, complementarities, and temporal dynamics raised in this review. Specifically, studies can test claims that without system-level political capacity (i.e., trust in government, accountability, legitimacy), having high operational and analytical capacities at individual and/or organizational levels may have less impact on design since mobilization of these capacities might not deliver legitimate, widely supported policies at later stages of policy design. Forthcoming research could also explore whether or not system-level political capacity is indeed the most fundamental type of capacity, while the remaining are more secondary or complementary. It may also be the case that any ‘secondary’ capacities at individual or managerial levels can be observed to contribute to solidifying political capacity at system level, and research on these directional relationships between different orders of policy capacity would greatly enrich the discussion on policy process and more specifically policy design.

These questions reveal a certain degree of agitation and urgency with wanting to find critical answers about how to match publicly salient goals with means that are effective, durable, equitable and also flexible in erratic policy contexts. Joining together concerns about capacity and how to design policy answers effectively signifies a promising, and perhaps also a vital avenue of further academic enquiry, and especially so in times marked by unprecedented public crises.

Acknowledgements

The authors thank Kidjie Saguin for his support in preparation of earlier versions of this paper. Kerem gratefully acknowledges the organizational support of Sabanci University and GLODEM, Koc University, as part of the paper was written during his Part-time lectureship at Sabanci University.

Availability of data and material

Code availability, declarations.

The authors declare that they have no conflict of interest.

1 We thank an anonymous referee for raising this essential point.

2 We are aware of two major caveats. Firstly, our database only covers journal articles written in English. In addition, our database excluded monographs and edited book chapters. Studies that are written in other languages and those published as monographs and edited book chapters are likely to offer additional insights to the findings in the article, which demands further research.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Contributor Information

Ishani Mukherjee, Email: gs.ude.ums@minahsi .

M. Kerem Coban, Email: [email protected] .

Azad Singh Bali, Email: [email protected] .

• Alaerts GJ. Adaptive policy implementation: Process and impact of Indonesia’s national irrigation reform 1999–2018. World Development. 2020; 129 :104880. [ Google Scholar ]
• Anderson JE. Public policymaking. Praeger; 1975. [ Google Scholar ]
• Ansell C, Gash A. Collaborative governance in theory and practice. Journal of Public Administration Research and Theory. 2008; 18 (4):543–571. [ Google Scholar ]
• Ansell C, Trondal J, Øgård M, editors. Governance in turbulent times. Oxford University Press; 2016. [ Google Scholar ]
• Bajpai K, Chong B. India's foreign policy capacity. Policy Design and Practice. 2019; 2 (2):137–162. [ Google Scholar ]
• Bakır C, Çoban MK. How can a seemingly weak state in the financial services industry act strong? The role of organizational policy capacity in monetary and macroprudential policy. New Perspectives on Turkey. 2019; 61 :71–96. [ Google Scholar ]
• Bakvis H. Rebuilding policy capacity in the era of the fiscal divident: A report from Canada. Governance. 2002; 13 (1):71–103. [ Google Scholar ]
• Bali AS, Capano G, Ramesh M. Anticipating and designing for policy effectiveness. Policy and Society. 2019; 38 (1):1–13. [ Google Scholar ]
• Bali AS, Ramesh M. Designing effective healthcare: Matching policy tools to problems in China. Public Administration and Development. 2017; 37 (1):40–508. [ Google Scholar ]
• Bali AS, Ramesh M. Policy capacity: A design perspective. In: Howlett M, Mukherjee I, editors. Routledge handbook of policy design. Routledge; 2018. pp. 331–344. [ Google Scholar ]
• Bali AS, Ramesh M. Assessing health reform: Studying tool appropriateness and critical capacities. Policy and Society. 2019; 38 (1):148–166. [ Google Scholar ]
• Bali AS, Ramesh M. Governing healthcare in India: A policy capacity perspective. International Review of Administrative Sciences. 2021 doi: 10.1177/00208523211002605. [ CrossRef ] [ Google Scholar ]
• Banerjee AV, Bardhan PK, Somanathan R, editors. Poverty and income distribution in India. Juggernaut; 2017. [ Google Scholar ]
• Bardhan P. State and development: The need for a reappraisal of the current literature. Journal of Economic Literature. 2016; 54 (3):862–892. [ Google Scholar ]
• Baskoy T, Evans B, Shields J. Assessing policy capacity in Canada’s public services: Perspectives of deputy and assistant deputy ministers. Canadian Public Administration. 2011; 54 (2):217–234. [ Google Scholar ]
• Bengston DN, Fletcher JO, Nelson KC. Public policies for managing urban growth and protecting open space: Policy instruments and lessons learned in the United States. Landscape and Urban Planning. 2004; 69 (2–3):271–286. [ Google Scholar ]
• Bernstein S, Cashore B. Complex global governance and domestic policies: Four pathways of influence. International Affairs. 2012; 88 (3):585–604. [ Google Scholar ]
• Bobrow DB. Policy design: Ubiquitous, necessary and difficult. In: Peters BG, Pierre J, editors. Handbook of public policy. Sage; 2006. pp. 75–96. [ Google Scholar ]
• Bobrow DB, Dryzek JS. Policy analysis by design. University of Pittsburgh Press; 1987. [ Google Scholar ]
• Borrás S. Policy learning and organizational capacities in innovation policies. Science and Public Policy. 2011; 38 (9):725–734. [ Google Scholar ]
• Bovens M, ‘t Hart P. Revisiting the study of policy failures. Journal of European Public Policy. 2016; 23 (5):653–666. [ Google Scholar ]
• Braathen NA. Instrument mixes for environmental policy: How many stones should be used to kill a bird? International Review of Environmental and Resource Economics. 2007; 1 (2):185–235. [ Google Scholar ]
• Braathen NA, Croci E. Environmental agreements used in combination with other policy instruments. In: Croci E, editor. The handbook of environmental voluntary agreements: Design, implementation and evaluation issues. Springer; 2005. pp. 335–364. [ Google Scholar ]
• Braun D. Organising the political coordination of knowledge and innovation policies. Science and Public Policy. 2008; 34 (4):227–239. [ Google Scholar ]
• Braun D. Lessons on the political coordination of knowledge and innovation policies. Science and Public Policy. 2008; 34 (4):289–298. [ Google Scholar ]
• Briassoulis H. Policy integration for complex environmental problems. Ashgate; 2005. [ Google Scholar ]
• Brown ML. Scientific uncertainty ad learning in European Union environmental policymaking. Policy Studies Journal. 2000; 28 (3):576–596. [ Google Scholar ]
• Busuioc EM, Lodge M. The reputational basis of public accountability. Governance. 2016; 29 (2):247–263. [ Google Scholar ]
• Capano G. Policy design spaces in reforming governance in higher education: The dynamics in Italy and the Netherlands. Higher Education. 2018; 75 (4):675–694. [ Google Scholar ]
• Capano G, Howlett H, Jarvis DSL, Ramesh M, Goyal N. Mobilizing policy (in)capacity to fight COVID-19: Understanding variations in national responses. Policy and Society. 2020; 39 (3):285–308. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Capano, G., & Howlett, M. (2020). The knowns and unknowns of policy instrument analysis: Policy tools and the current research agenda on policy mixes. Sage Open . 10.1177/2158244019900568.
• Capano G, Pavan E. Designing anticipatory policies through the use of ICTs. Policy and Society. 2019; 38 (1):96–117. [ Google Scholar ]
• Capano G, Woo JJ. Resilience and robustness in policy design: A critical appraisal. Policy Sciences. 2017; 50 (3):399–426. [ Google Scholar ]
• Cashore B, Bernstein S, Humphreys D, Visseren-Hamakers I, Rietig K. Designing stakeholder learning dialogues for effective global governance. Policy and Society. 2019; 38 (1):118–147. [ Google Scholar ]
• Chindarkar N, Howlett M, Ramesh M. Introduction to the special issue: Conceptualizing effective social policy design: Design spaces and capacity challenges. Public Administration and Developmentv. 2017; 37 (1):3–14. [ Google Scholar ]
• Clarke A, Craft J. The vestiges and vanguards of policy design in a digital context. Canadian Public Administration. 2017; 60 (4):476–497. [ Google Scholar ]
• Coban MK. Diffuse interest groups and regulatory policy change: Financial consumer protection in Turkey. Interest Groups an Advocacy. 2020; 9 (2):220–243. [ Google Scholar ]
• Coban MK. Compliance forces, domestic policy process, and international regulatory standards: Compliance with Basel III. Business and Politics. 2020; 22 (1):161–195. [ Google Scholar ]
• Compton M, ‘t Hart P. How to ‘see’ great policy successes. In: Compton M, ‘t Hart P, editors. Great policy successes. Oxford University Press; 2019. pp. 1–20. [ Google Scholar ]
• Considine M. Thinking outside the box? Applying design theory to public policy. Politics and Policy. 2012; 40 (4):704–724. [ Google Scholar ]
• Craft J, Daku M. A comparative assessment of elite policy recruits in Canada. Journal of Comparative Policy Analysis: Research and Practice. 2017; 19 (3):207–226. [ Google Scholar ]
• Dare L. From global forests to local politics: Unwrapping the boundaries within forest certification. Australian Journal of Political Science. 2018; 53 (4):529–547. [ Google Scholar ]
• Daugbjerg C, Fraussen B, Halpin D. Interest group and policy capacity: Models of engagement, policy goods and networks. In: Wu X, Howlett M, Ramesh M, editors. Policy capacity and governance: Assessing governmental competences and capabilities in theory and practice. Springer; 2018. pp. 243–261. [ Google Scholar ]
• del Río P. Analysing the interactions between renewable energy promotion and energy efficiency support schemes: The impact of different instruments and design elements. Energy Policy. 2010; 38 (9):4978–4989. [ Google Scholar ]
• del Río, P., & Howlett, M. (2013). “Beyond the ‘Tinbergen rule’ in policy design: Matching tools and goals in policy portfolios.”SSRN Scholarly Paper. Available at https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2247238 . Last accessed 11 June 2020).
• Donahue AK, Selden SC, Ingraham PW. Measuring government management capacity: A comparative analysis of city human resources management systems. Journal of Public Administration Research and Theory. 2000; 10 (2):381–412. [ Google Scholar ]
• Doremus H. A policy portfolio approach to biodiversity protection on private lands. Environmental Science & Policy. 2003; 6 (3):217–232. [ Google Scholar ]
• Dorsch MJ, Flachsland C. A polycentric approach to global climate governance. Global Environmental Politics. 2017; 17 (2):45–64. [ Google Scholar ]
• Douglas, S., Hart, P. T., Ansell, C., Anderson, L., Flinders, M., Head, B., & Moynihan, D. (2019). Towards positive public administration: A manifesto . Available online at https://www.researchgate.net/publication/336362499_Towards_Positive_Public_Administration_A_Manifesto .
• Dunlop CA. Organizational political capacity as learning. Policy and Society. 2015; 34 (4–5):259–270. [ Google Scholar ]
• Giest S. Big data for policymaking: Fad or fasttrack? Policy Sciences. 2017; 50 (3):367–382. [ Google Scholar ]
• Giest S, Mukherjee I. Behavioral instruments in renewable energy and the role of big data: A policy perspective. Energy Policy. 2018; 123 :360–366. [ Google Scholar ]
• Gilabert P, Lawford-Smith H. Political feasibility: A conceptual exploration. Political Studies. 2012; 60 (4):809–825. [ Google Scholar ]
• Golan J, Sicular T, Umapathi N. Unconditional cash transfers in China: Who benefits from the rural minimum living standard guarantee (Dibao) program? World Development. 2017; 93 :316–336. [ Google Scholar ]
• Grabosky PN. Green markets: Environmental regulation by the private sector. Law and Policy. 1994; 16 (4):419–448. [ Google Scholar ]
• Grant W. Policy instruments in the common agricultural policy. West European Politics. 2010; 33 (1):22–38. [ Google Scholar ]
• Gunningham N, Grabosky PN, Sinclair D. Smart regulation: Designing environmental policy. Clarendon Press; 1998. [ Google Scholar ]
• Haas PM. Introduction: Epistemic communities and international policy coordination. International Organization. 1992; 46 (1):1–35. [ Google Scholar ]
• Hartley J, Alford J, Hughes O, Yates S. Public value and political astuteness in the work of public managers: The art of the possible. Public Administration. 2015; 93 (1):195–211. [ Google Scholar ]
• Hartley K, Zhang J. Measuring policy capacity through governance indices. In: Wu X, Howlett M, Ramesh M, editors. Policy capacity and governance: Assessing governmental competences and capabilities in theory and practice. Springer; 2018. pp. 67–97. [ Google Scholar ]
• Hood C. Intellectual obsolescence and intellectual makeovers: Reflections on the tools of government after two decades. Governance. 2007; 20 (1):127–144. [ Google Scholar ]
• Hoppe R. Institutional constraints and practical problems in deliberative and participatory policy making. Policy and Politics. 2011; 39 (2):163–186. [ Google Scholar ]
• Hoppe R. Heuristics for practitioners of design: Rule-of-thumb for structuring unstructured problems. Public Policy and Administration. 2018; 33 (4):384–408. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Howlett M. Managing the “hollow state”: Procedural policy instruments and modern governance. Canadian Public Administration. 2000; 43 (4):412–431. [ Google Scholar ]
• Howlett M. Governance modes, policy regimes and operational plans: A multi-level nested model of policy instrument choice and policy design. Policy Sciences. 2009; 42 (1):73–89. [ Google Scholar ]
• Howlett M. Policy analytical capacity and evidence-based policymaking: Lessons from Canada. Canadian Public Administration. 2009; 52 (2):153–172. [ Google Scholar ]
• Howlett M. Designing public policies: Principles and instruments. Routledge; 2011. [ Google Scholar ]
• Howlett M. From the ‘old’ to the ‘new’ policy design: Design thinking beyond markets and collaborative governance. Policy Sciences. 2014; 47 (3):187–207. [ Google Scholar ]
• Howlett M. Policy design: What, who, how and why? In: Halpern C, Lascoumes P, Patrick LG, editors. L’instrumentation et ses effets. Presses de Sciences Po; 2014. pp. 281–315. [ Google Scholar ]
• Howlett M. Policy analytical capacity: The supply and demand for policy analysis in government. Policy and Society. 2015; 34 (3–4):173–182. [ Google Scholar ]
• Howlett M. The criteria for effective policy design: Character and context in policy instrument choice. Journal of Asian Public Policy. 2018; 11 (3):245–266. [ Google Scholar ]
• Howlett M. Dealing with the dark side of policy-making: Managing behavioural risk and volatility in policy designs. Journal of Comparative Policy Analysis. 2020; 22 (6):612–625. [ Google Scholar ]
• Howlett M, Capano G, Ramesh M. Designing for robustness: Surprise, agility and improvisation in policy design. Policy and Design. 2018; 37 (4):405–421. [ Google Scholar ]
• Howlett M, Lejano RP. Tales from the Crypt: The rise and fall (and rebirth?) of policy design. Administration and Society. 2013; 45 (3):357–381. [ Google Scholar ]
• Howlett M, Mukherjee I. An Asian perspective on policy instruments: Policy styles, governance modes and critical capacity challenges. Asia Pacific Journal of Public Administration. 2016; 38 (1):24–42. [ Google Scholar ]
• Howlett M, Mukherjee I. The importance of policy design: Effective processes, tools and outcomes. In: Howlett M, Mukherjee I, editors. Routledge handbook of policy design. Routledge; 2018. pp. 3–19. [ Google Scholar ]
• Howlett M, Mukherjee I. The contribution of comparative policy analysis to policy design: Articulating principles of effectiveness and clarifying design spaces. Journal of Comparative Policy Analysis. 2018; 20 (1):72–87. [ Google Scholar ]
• Howlett M, Mukherjee I, Rayner J. The elements of effective program design: A two-level analysis. In: Howlett M, Mukherjee I, editors. Handbook of policy formulation. Edward Elgar Publishing; 2017. pp. 129–144. [ Google Scholar ]
• Howlett M, Mukherjee I, Woo JJ. From tools to toolkits in policy design studies: The new design orientation towards policy formulation research. Policy and Politics. 2015; 43 (2):291–311. [ Google Scholar ]
• Howlett M, Ramesh M. The two orders of governance failure: Design mismatches and policy capacity issues in modern governance. Policy and Society. 2014; 33 (4):317–327. [ Google Scholar ]
• Howlett M, Ramesh M. Achilles’ heels of governance: Critical capacity deficits and their role in governance failures. Regulation and Governance. 2015; 10 (4):301–313. [ Google Scholar ]
• Howlett M, Ramesh M, Capano G. Policy-makers, policy-takers and policy tools: Dealing with behavioural issues in policy design. Journal of Comparative Policy Analysis. 2020; 22 (6):487–497. [ Google Scholar ]
• Howlett M, Ramesh M, Perl A. Studying public policy: Principles and processes. 4. Oxford University Press; 2020. [ Google Scholar ]
• Howlett M, Ramesh M, Wu X. Understanding the persistence of policy failures: The role of politics, governance and uncertainty. Public Policy and Administration. 2015; 30 (3–4):209–220. [ Google Scholar ]
• Howlett M, Rayner J. Design principles for policy mixes: Cohesion and coherence in ‘new governance arrangements’ Policy and Society. 2007; 26 (4):1–18. [ Google Scholar ]
• Howlett M, Rayner J. Patching vs packaging in policy formulation: Assessing policy portfolio design. Politics and Governance. 2013; 1 (2):170–182. [ Google Scholar ]
• Howlett M, Tosun J, editors. Routledge handbook of policy styles. Routledge; 2021. [ Google Scholar ]
• Huitema D, Jordan A, Munaretto S, Hildén M. Policy experimentation: Core concepts, political dynamics, governance and impacts. Policy Sciences. 2018; 51 (2):143–159. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Jensen O. Designing effective water policy: Capacity and effectiveness of reforms in developing countries. Policy and Society. 2018; 38 (1):77–95. [ Google Scholar ]
• Jordan A, Benson D, Wurzel R, Zito A. Policy instruments in practice. In: Dryzek JS, Norgaard RB, Schlosberg D, editors. Oxford handbook of climate change and society. Oxford University Press; 2011. pp. 537–549. [ Google Scholar ]
• Jordan A, Benson D, Wurzel R, Zito A. Environmental policy: Governing by multiple policy instruments? In: Richardson JJ, editor. Constructing a policy state? Policy dynamics in the EU. Oxford University Press; 2012. pp. 105–124. [ Google Scholar ]
• Jordan A, Huitema D. Innovations in climate policy: The politics of invention, diffusion, and evaluation. Environmental Politics. 2014; 23 (5):715–734. [ Google Scholar ]
• Karo E, Kattel R. Innovation and the state: Towards an evolutionary theory of policy capacity. In: Wu X, Howlett M, Ramesh M, editors. Policy capacity and governance: Assessing governmental competences and capabilities in theory and practice. Springer; 2018. pp. 123–150. [ Google Scholar ]
• Kelemen RD. Europe’s other democratic deficit: National authoritarianism in Europe’s Democratic Union. Government and Opposition. 2017; 52 (2):211–238. [ Google Scholar ]
• Kimbell L, Vesnić-Alujević L. After the toolkit: Anticipatory logics and the future of government. Policy Design and Practice. 2020; 3 (2):95–108. [ Google Scholar ]
• Lang A. Government capacities and stakeholders: What facilitates ehealth legislation? Globalization and Health. 2014; 10 (4):1–14. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Leong Ch, Howlett M. Theorizing the behavioral state: Resolving the theory-practice paradox of policy sciences. Public Policy and Administration. 2020 doi: 10.1177/0952076720977588. [ CrossRef ] [ Google Scholar ]
• Linder SH, Peters BG. From social theory to policy design. Journal of Public Policy. 1984; 4 (3):237–259. [ Google Scholar ]
• Lindquist EA. Public managers and policy communities: Learning to meet new challenges. Canadian Public Administration. 1992; 35 (2):127–159. [ Google Scholar ]
• Lodge M. Crisis, resources and the state: Executive politics in the age of the depleted state. Political Studies Review. 2013; 11 (3):378–390. [ Google Scholar ]
• Lodhi, I. (2018). Policy capacity: Introducing a comprehensive index. Paper prepared for the international workshop on public policy—Pittsburgh 2018 , June 26–28, Pittsburgh, United States.
• Luetjens J, Mintrom M, ‘t Hart P, editors. Successful public policy: Lessons from Australia and New Zealand. Australian National University Press; 2019. [ Google Scholar ]
• Maerz S, et al. State of the world 2019: Autocratization surges–resistance grows. Democratization. 2020; 27 (6):909–927. [ Google Scholar ]
• Majone G. On the notion of political feasibility. European Journal of Political Research. 1975; 3 (2):259–274. [ Google Scholar ]
• Maurya D, Ramesh M. Program design, implementation and performance: The case of social health insurance in India. Health Economics, Policy and Law. 2019; 14 (4):487–508. [ PubMed ] [ Google Scholar ]
• May PJ. Policy design and implementation. In: Peters BG, Pierre J, editors. Handbook of public administration. Sage Publications; 2003. pp. 223–233. [ Google Scholar ]
• May PJ. Regulation and compliance motivations: Examining different approaches. Public Administration Review. 2005; 65 (1):31–44. [ Google Scholar ]
• McConnell A. Policy success and failure. Oxford Research Encyclopedia of Politics. 2017 doi: 10.1093/acrefore/9780190228637.013.137. [ CrossRef ] [ Google Scholar ]
• McConnell A. Hidden agendas: Shining a light on the dark side of public policy. European Journal of Public Policy. 2018; 25 (12):1739–1758. [ Google Scholar ]
• McConnell A. The use of placebo policies to escape from policy traps. Journal of European Public Policy. 2020; 27 (7):957–976. [ Google Scholar ]
• Meckling J, Nahm J. The power of process: State capacity and climate policy. Governance. 2018; 31 (4):741–757. [ Google Scholar ]
• Menahem G, Stein R. High-capacity and low-capacity governance networks in welfare services delivery: A typology and empirical examination of the case of Israeli municipalities. Public Administration. 2013; 91 (1):211–231. [ Google Scholar ]
• Mergel I, Ganapati S, Whitford AB. Agile: A new way of governing. Public Administration Review. 2021; 81 (1):161–165. [ Google Scholar ]
• Montpetit É. Policy design for legitimacy: Expert knowledge, citizens, time and inclusion in the United Kingdom’s biotechnology sector. Public Administration. 2008; 86 (1):259–277. [ Google Scholar ]
• Mukherjee I, Bali AS. Policy effectiveness and capacity: Two sides of the design coin. Policy Design and Practice. 2019; 2 (2):103–114. [ Google Scholar ]
• Mukherjee I, Giest S. Designing policies in uncertain contexts: Entrepreneurial capacity and the case of the European emission trading scheme. Public Policy and Administration. 2019; 34 (3):262–286. [ Google Scholar ]
• Mukherjee I, Howlett M. An Asian perspective on policy instruments: Policy styles, governance modes and critical capacity challenges. AsiaPacific Journal of Public Administration. 2016; 38 (1):24–42. [ Google Scholar ]
• Mukherjee I, Mukherjee N. Designing for sustainable outcomes: Espousing behavioural change into co-production programmes. Policy and Society. 2018; 37 (3):326–346. [ Google Scholar ]
• Nair S, Howlett M. Policy myopia as a source of policy failure: Adaptation and policy learning under deep uncertainty. Policy and Politics. 2017; 45 (1):103–118. [ Google Scholar ]
• Newman J, Cherney A, Head BW. Policy capacity and evidence-based policy in the public service. Public Management Review. 2017; 19 (2):157–174. [ Google Scholar ]
• Norris P, Inglehart R. Cultural backlash: Trump, Brexit, and authoritarian populism. Cambridge University Press; 2019. [ Google Scholar ]
• O’Flynn J. Rethinking relationships: Clarity, contingency, and capabilities. Policy Design and Practice. 2019; 2 (2):115–136. [ Google Scholar ]
• Pal LA, Clark ID. Making reform stick: Political acumen as an element of political capacity for policy change and innovation. Policy and Society. 2015; 34 (3–4):247–257. [ Google Scholar ]
• Pestoff VA, Brandsen T, Verschuere B. New public governance, the third sector and co-production. Routledge; 2012. [ Google Scholar ]
• Peters BG. Policy capacity in public administration. Policy and Society. 2015; 34 (3–4):219–228. [ Google Scholar ]
• Peters BG. Can we be casual about being causal? Journal of Comparative Policy Analysis. 2020 doi: 10.1080/13876988.2020.1793327. [ CrossRef ] [ Google Scholar ]
• Peters BG, Eliadis P, Hill M, Howlett M. Conclusion: The future of instruments research. In: Eliadis P, Hill MM, Howlett M, editors. Designing government: From instruments to governance. McGill-Queen’s University Press; 2005. pp. 353–363. [ Google Scholar ]
• Peters BG, et al. Designing for policy effectiveness: Defining and understanding a concept. Cambridge University Press; 2018. [ Google Scholar ]
• Peters BG, Pierre J. Governance and policy problems: Instruments as unitary and mixed modes of policy intervention. Asia Pacific Journal of Public Administration. 2015; 37 (4):224–235. [ Google Scholar ]
• Rahman M, Wellstead A, Howlett M. From adaptive capacity to policy capacity adaptation assessments: Insights from the policy sciences. In: Keskitalo ECH, Preston BL, editors. Research handbook on climate change adaptation policy. Edward Elgar; 2019. pp. 291–309. [ Google Scholar ]
• Ramesh M, Bali AS. The remarkable healthcare performance in Singapore. In: Compton M, ‘t Hart P, editors. Great Policy Successes. Oxford University Press; 2019. pp. 42–62. [ Google Scholar ]
• Rayner J, McNutt K, Wellstead AM. Dispersed capacity and weak coordination: The challenge of climate change adaptation in Canada’s forest policy sector. Review of Policy Research. 2013; 30 (1):66–90. [ Google Scholar ]
• Rimkuté D. Organizational reputation and risk regulation: The effect of reputational threats on agency scientific outputs. Public Administration. 2018; 96 (1):70–83. [ Google Scholar ]
• Rotberg RI. Good governance means performance and results. Governance. 2014; 27 (3):511–518. [ Google Scholar ]
• Saguin K, Ramesh M, Howlett M. Policy work and capacities in a developing country: Evidence from the Philippines. Asia Pacific Journal of Public Administration. 2018; 40 (1):1–22. [ Google Scholar ]
• Schout A, Andrew J. The European Union’s governance ambitions and its administrative capacities. Journal of European Public Policy. 2008; 15 (4):957–974. [ Google Scholar ]
• Sewerin S, Béland D, Cashore B. Designing policy for the long term: Agency, policy feedback and policy change. Policy Sciences. 2020; 53 (2):243–252. [ Google Scholar ]
• Sidney MS. Policy formulation: Design and tools. In: Fischer F, Miller GJ, Sidney MS, editors. Handbook of public policy analysis: Theory, politics and methods. CRC Taylor & Francis; 2007. pp. 79–87. [ Google Scholar ]
• Siwale J, Okoye O. Microfinance regulation and social sustainability of microfinance institutions: The case of Nigeria and Zambia. Annals of Public and Cooperative Economics. 2017; 88 (4):611–632. [ Google Scholar ]
• Skeete J. Examining the role of policy design and policy interaction in EU automotive emissions performance gaps. Energy Policy. 2017; 104 :373–381. [ Google Scholar ]
• Skodvin T, Gullberg AT, Aakre S. Target-group influence and political feasibility: The case of climate policy design in Europe. Journal of European Public Policy. 2010; 17 (6):854–873. [ Google Scholar ]
• Spendzharova AB. Regulatory cascading: Limitations of policy design in European banking structural reforms. Policy and Society. 2016; 35 (3):227–237. [ Google Scholar ]
• Stazyk EC, Goerdel HT. The benefits of bureaucracy: Public managers’ perceptions of political support, goal ambiguity, and organizational effectiveness. Journal of Public Administration Research and Theory. 2011; 21 (4):645–672. [ Google Scholar ]
• Stirling A. Keep it complex. Nature. 2010; 468 (7327):1029–1031. [ PubMed ] [ Google Scholar ]
• Stone D. The “knowledge bank” and the global development network. Global Governance. 2003; 9 (1):43–61. [ Google Scholar ]
• Stone D, Ladi S. Global public policy and transnational administration. Public Administration. 2015; 93 (4):839–855. [ Google Scholar ]
• The Economist. (2020). Countries are using apps and data networks to keep tabs on the pandemic . March 26.
• Thelen K. How institutions evolve: The political economy of skills in Germany, Britain, the United States, and Japan. Cambridge University Press; 2004. [ Google Scholar ]
• Tiernan, A., & Wanna, J. (2006). Competence, capacity, capability: Towards conceptual clarity in the discourse of declining policy skills. Paper presented at the govnet international conference . Australian National University, Canberra, Australia.
• Tosun J, Lang A. Policy integration: Mapping the different concepts. Policy Studies. 2017; 38 (6):553–570. [ Google Scholar ]
• Van Der Steen M, Scherpenisse J, Van Twist M. Anticipating surprise: The case of the early warning system of Rijkswaterstaat in the Netherlands. Policy and Society. 2018; 37 (4):473–490. [ Google Scholar ]
• Vince J, Nursey-Bray M. Policy capacity in oceans governance: Rio+20 and Australia’s outcomes. Australian Journal of Maritime and Ocean Affairs. 2016; 8 (3):165–179. [ Google Scholar ]
• Virani A. Notions of policy effectiveness and implications for policy design: Insights from public–private partnerships in India. Policy Design and Practice. 2019; 2 (2):198–214. [ Google Scholar ]
• von Tunzelmann N. Technology and technology policy in the postwar UK: Market failure or“network failure?” Revue D’économie Industrielle. 2010; 129–130 :237–258. [ Google Scholar ]
• Walter J, et al. Policy capacity in disruptive times. Australian Journal of Political Science. 2020; 55 (1):72–85. [ Google Scholar ]
• Weaver, K. (2009a). If you build it, will they come? Overcoming unforeseen obstacles to program effectiveness . The Tansley Lecture, University of Saskatchewan.
• Weaver K. Target compliance: The final frontier of policy implementation. Brookings Institution Press; 2009. [ Google Scholar ]
• Weber EP, Lovrich NP, Gaffney MJ. Assessing collaborative capacity in a multidimensional world. Administration and Society. 2007; 39 (2):194–220. [ Google Scholar ]
• Weible CM, et al. COVID-19 and the policy sciences: Initial reactions and perspectives. Policy Sciences. 2020; 53 (2):225–241. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Wellstead AM, Stedman RC, Howlett M. Policy analytical capacity in changing governance contexts: A structural equation model (SEM) study of contemporary Canadian policy work. Public Policy and Administration. 2011; 26 (3):353–373. [ Google Scholar ]
• Wen Z. Government purchase of services in China: Similar intentions, different policy designs. Public Administration and Development. 2017; 37 (1):65–78. [ Google Scholar ]
• Williams RA, McNutt K. Climate change adaptation and policy capacity in the Canadian finance sector: A meso analysis. Review of Policy Research. 2013; 30 (1):91–113. [ Google Scholar ]
• Woo JJ. Policy capacity and Singapore’s response to the COVID-19 pandemic. Policy and Society. 2020; 39 (3):345–362. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Woo JJ, Ramesh M, Howlett M. Legitimation capacity: System-level resources and political skills in public. Policy and Society. 2015; 34 (3–4):271–283. [ Google Scholar ]
• Woo JJ, Ramesh M, Howlett M, Coban MK. Dynamics of global financial governance: Constraints, opportunities, and capacities in Asia. Policy and Society. 2016; 35 (3):269–282. [ Google Scholar ]
• Wu X, Ramesh M, Howlett M. Policy capacity: A conceptual framework for understanding policy competences and capabilities. Policy and Society. 2015; 34 (3–4):165–171. [ Google Scholar ]
• Wu X, Ramesh M, Howlett M. Policy capacity: Conceptual framework and essential components. In: Wu X, Howlett M, Ramesh M, editors. Policy capacity and governance: Assessing governmental competences and capabilities in theory and practice. Springer; 2018. pp. 1–25. [ Google Scholar ]
• Wu X, Ramesh M, Howlett M, Fritzen S. The public policy primer: Managing public policy. Routledge; 2010. [ Google Scholar ]
• Xiarchogiannopoulou E. Reform technology and consensus in building in discursive institutionalism: The integration of knowledge in policy formulation in Greek pension reform, 1990–2002. Critical Policy Studies. 2015; 9 (4):416–433. [ Google Scholar ]
• Yan Y, Saguin K. Policy capacity matters for capacity development. International Review of Administrative Sciences. 2021 doi: 10.1177/0020852320983867. [ CrossRef ] [ Google Scholar ]
• Yee W, Liu P. Governance capacity and regulatory enforcement: The case of food safety regulatory reform in Beijing. International Review of Administrative Sciences; 2021. [ Google Scholar ]
• Yi H, Feiock RC. Policy tool interactions and the adoption of state renewable portfolio standards. Review of Policy Research. 2012; 29 (2):193–206. [ Google Scholar ]

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 12 December 2017

Rethinking policy ‘impact’: four models of research-policy relations

• Christina Boswell 1 &
• Katherine Smith 1  

Palgrave Communications volume  3 , Article number:  44 ( 2017 ) Cite this article

46k Accesses

94 Citations

114 Altmetric

Metrics details

• Politics and international relations
• Science, technology and society
• Social policy

A Correction to this article was published on 20 February 2018

This article has been updated

Political scientists are increasingly exhorted to ensure their research has policy ‘impact’, most notably via Research Excellence Framework (REF) impact case studies, and ‘pathways to impact’ statements in UK Research Council funding applications. Yet the assumptions underpinning these frameworks often fail to reflect available evidence and theories. Notions of ‘impact’, ‘engagement’ and ‘knowledge exchange’ are typically premised on simplistic, linear models of the policy process, according to which policy-makers are keen to ‘utilise’ expertise to produce more ‘effective’ policies. Such accounts overlook the rich body of literature in political science, policy studies, and sociology of knowledge, which offer more complex and nuanced accounts. Drawing on this wider literature, this paper sets out four different approaches to theorising the relationship: (1) knowledge shapes policy; (2) politics shapes knowledge; (3) co-production; and (4) autonomous spheres. We consider what each of these four approaches suggests about approaches to incentivising and measuring research impact.

Similar content being viewed by others

Mapping the community: use of research evidence in policy and practice

Co-producing the science–policy interface: towards common but differentiated responsibilities

Interdisciplinary research attracts greater attention from policy documents: evidence from COVID-19

Introduction.

The new research ‘impact’ agenda is likely to have a profound effect on the social science research community in wide-ranging ways, shaping the sorts of research questions and methods scholars are selecting, their networks and collaborations, as well as changing institutional structures of support within higher education institutions. Yet concepts and models for defining and measuring impact have been subject to surprisingly little social scientific scrutiny. While there is an extensive literature on research-policy relations across fields of social science (notably in sociology, science and technology studies, social policy, political science and public management), only a very narrow range of these contributions have been marshalled to develop guidance and practice on ‘impact’. Indeed, prevalent guidelines and models are frequently based on surprisingly simple and linear ideas about how research can be ‘utilised’ to produce more effective policies (Smith and Stewart, 2016 ).

In this article, we seek to advance the debate on impact by setting out four different approaches to theorising research-policy relations, drawn from wider social science literature. Each set of theories is categorised according to its core assumptions about the inter-relations between the two spheres. The first approach focuses on a ‘supply’ model of research-policy relations, examining how knowledge and ideas shape policy. The second challenges the idea that research is independent of politics and policy, instead focusing on how political power shapes knowledge. The third approach takes this line further, suggesting that research knowledge and governance are co-produced through an ongoing process of mutual constitution. And the fourth approach offers a radically contrasting account, suggesting that there is no overarching causality between science and politics, but that politics only selectively appropriates and gives meaning to scientific findings. Figure  1 offers a simple representation of these four ways of modelling the relations.

Research-policy relations

This figure represents in visual form the direction of influence between research, expert knowledge and science; and policy and politics. The first panel represents theories assuming that research shapes policy. The second panel depicts the idea that policy and politics shape the production of research. In the third panel, the circular arrows convey the idea of research and policy being mutually constitutive. While the fourth panel suggests that there is no direct causal relationship between research and policy, but that instead, the two ‘systems’ only selectively pick up on signals from the other system.

This four-way schema offers a useful resource in two main ways. First, it offers a classificatory tool for mapping, comparing and analysing a range of often disparate theoretical approaches in the emerging field of knowledge-policy relations–theories that emanate from a wide set of social science disciplines, and are informed by quite divergent assumptions about knowledge and governance. The second, more applied, use of the schema is to identify the plurality of ways of conceptualising knowledge-policy relations. In doing so, we demonstrate that prevalent models of impact are based on one particular set of assumptions about the role of research in policy, and not necessarily the most theoretically sophisticated at that. By briefly setting out each of the four sets of theories, we show how each is based on quite distinct assumptions about knowledge and policy, and that each has different implications for how we might go about defining and measuring impact.

The ‘impact’ agenda in UK research funding

The emphasis on ‘research impact’ has been increasing steadily across a number of OECD countries over the past decade, notably Australia (Donovan, 2008 ; Chubb and Watermeyer, 2016 ), Canada (Canadian Academy of Health Sciences CAHS, 2009 ), the Netherlands (Mostert et al., 2010 ) and the USA (Grant et al., 2010 ) but the influence of this agenda is particularly pronounced in the UK, which can be seen as something of a pioneer in implementing these approaches (see Bornmann, 2013 and Grant et al., 2010 for useful comparative overviews). There are currently two major incentives for social scientists in the UK to demonstrate that their research influences policy. First, the national appraisal mechanism for assessing university research (which informs decisions about the distribution of core research funding), known as the Research Excellence Framework (REF), has begun awarding 20% of overall scores to institutions on the basis of case studies of research impact (UK higher education funding bodies 2011 ). Second, accounts of the work that will be undertaken to achieve research impact (‘pathways to impact’) now form a significant section of grant application processes for the UK funding councils (Research Councils UK, Undated ). The upshot is that obtaining core research funding and project-specific grants from publicly funded sources in the UK are now strongly dependent on researchers’ abilities to respond adequately to questions about the non-academic value of their work (Smith and Stewart, 2016 ).

The current focus on ‘research impact’ reflects a longer-standing concern with the societal return on public funding of science (Brewer, 2011 ; Clarke, 2010 ). This agenda was given particular impetus by New Labour government commitments to taking a more ‘evidence-based’ approach to policymaking (Labour Party, 1997 ), with official statements evoking a simple, linear conceptualisation of the relationship between research and policy (e.g., Cabinet Office, 1999 , 2000 ; Blunkett, 2000 ). It is this kind of thinking that appears to have shaped tools and guidance on impact (Smith, 2013a ). Indeed, while different public bodies have adopted a variety of models, RCUK and REF advisory documents tend to share a number of common features (AHRC, 2014 , 2015 ; ESRC, 2014a , 2014b , 2014c ; MRC, 2014 ; Research Councils UK, Undated): (i) a consensus that researchers have a responsibility to articulate the impact of their research to non-academic audiences; (ii) an assumption (most explicit in the REF impact case studies) that this impact can be documented and measured; (iii) a belief that the distribution of research funding should (at least to some extent) reflect researchers’ ability to achieve ‘impact’; and, following from this, (iv) an expectation that researchers’ own efforts to achieve research impact will play a significant role in explaining why some research has impact beyond academia and some does not.

This approach is exemplified in HEFCE’s template for REF2014 impact ‘case studies’ (REF, 2014, 2011 ). The template calls for an account of the ‘underpinning research’ that exerted impact, implying that impact is achieved through policy-makers adjusting their beliefs in response to clearly delineated research findings. The implication is that research findings are created independently of policy or politics: research is treated as an exogenous variable that feeds into policy-making. Secondly, such findings are expected to have been published as ‘outputs’ that are rated 2*, or ‘nationally leading in terms of their originality, significance and rigour’ (REF2014, 2014 ). Thus a clear link is posited between the quality of research and the desirability of rewarding impact: impactful research should meet a certain quality threshold. Thirdly, researchers are required to chart how their findings came to exert impact, and to provide evidence to corroborate their claims. Evocative of the ‘pathways to impact’ section of RCUK grant proposals (Research Councils UK, Undated ), this requirement implies that researchers can trace the effects of their work through describing a series of concrete activities and information flows – events, meetings, media coverage, and so on.

There is currently no agreed way of tracking research impacts and, in this context, some academics have identified more specific frameworks and approaches, including the ‘payback framework’ (Donovan and Hanney, 2011 ) and the ‘research contribution framework’ (Morton, 2015 ). However, others have criticised the simplistic and linear conceptualisations of research-policy relations that appear to underpin the UK’s overarching approach to research impact, particularly those with in-depth knowledge of the policy process and/or the relationship between research and policy (Greenhalgh et al., 2016 ; Smith and Stewart, 2016 ). Theories of public policy have shown that policy-making rarely occurs in such neat sequential stages (Cairney, 2016 ), and that evidence often plays a rather limited role in decision-making (Boswell, 2009a ). In the context of such criticisms and concerns, we consider the rich body of literature from political science, policy studies, sociology of knowledge, and science and technology studies, which has informed understandings of the complex relationship between knowledge and policy. Drawing on this wider literature, we now set out four different approaches to theorising the relationship, and consider their implications for the impact agenda.

Four approaches to conceptualising research-policy relations

Knowledge shapes policy.

A range of theories and models of the relationship between academic knowledge and policy were developed by US and UK scholars in the 1970s and 1980s (Blume, 1977 ; Caplan, 1979 ; Rein, 1980 ; Weiss, 1977 , 1979 ). Notably, a number of contributions produced ‘instrumental’ models of knowledge utilisation (see Weiss, 1979 for an overview), according to which knowledge either ‘drives’ policy, or policy problems stimulate research to provide direct solutions (again, see Weiss, 1979 ). Much of the work undertaken in the 1970s and 1980s demonstrated that while there are occasional examples of research feeding into policy in this manner, such simple models failed to capture the intricacies of the interactions between research and policy (Rein, 1980 ; Weiss, 1979 ). Yet, it was precisely these simple, instrumental notions of the role of research in policy that seem to have become increasingly embedded within UK policy, including higher education policy, leading Parsons to reflect that the Labour government’s commitments to ‘evidence-based policymaking’ marked:

not so much a step forward as a step backwards: a return to the quest for a positivist yellow brick road leading to a promised policy dry ground-somewhere, over Charles Lindblom - where we can know ‘what works’ and from which government can exercise strategic guidance. (Parsons, 2002 , p 45)

Understandably, official commitments to employing evidence in a direct, linear sense triggered a raft of assessments of the extent to which particular policies do reflect the available evidence. Perhaps unsurprisingly, most of these found the government’s use of evidence has been highly selective (e.g., Boswell 2009a , 2009b ; Katikireddi et al., 2011 ; Naughton, 2005 ; Stevens, 2007 ) and this, in turn, has triggered renewed interest in two, more complex models of the ways in which research knowledge shapes policy, each of which has very different implications for the research impact agenda.

The first of these approaches seeks to address what is perceived as a ‘gap’ between the research and policy communities. On this account, research has the potential to be highly relevant to policy, but its impact is often reduced by problems of communication. Research may not be disseminated in a form that is relevant or accessible to policy-makers; or officials have insufficient resources to process and apply research findings. For example, Lomas ( 2000 ) and Lavis ( 2006 ) both underline the importance of achieving shared understandings between researchers and policymakers, arguing that increased interaction between the two groups will improve the use of research in policy. These authors tend to assume that research would be more frequently employed by policymakers if only they could better access and understand the findings and if the findings were of relevance. Thus the focus is on improving the mechanisms of communication, and the levels of trust, between researchers and policymakers. A stronger version of this ‘gap’ account posits that this reflects a deeper cultural gap between researchers and policy actors. Thus Caplan ( 1979 ) suggests that these actors should be seen as distinct ‘communities’ guided by different values and beliefs–a notion we discuss further in the fourth set of theories, considered later in the paper.

The weaker version of this ‘gap’ approach, however, suggests that there are various practical steps that can be taken to improve the flow of knowledge from research to policy. Indeed, several reviews of knowledge transfer provide practical recommendations for researchers seeking to influence policy (Contandriopoulos et al., 2010 ; Innvaer et al., 2002 ; Mitton et al., 2007 ; Nutley et al., 2003 ; Oliver et al., 2013 ; Walter et al., 2005 ), suggesting researchers should ensure research is accessible, by providing clear, concise, timely summaries of the research, tailored to appropriate audiences; and develop ongoing, collaborative relationships with potential users to increase levels of trust and shared definitions of policy problems and responses. In structural terms, the findings of these reviews call for improved communication channels, via ‘knowledge broker’ roles and/or knowledge transfer training and sufficiently high incentives for researchers and research users to engage in knowledge exchange. Of the various conceptualisations of the relationship between research knowledge and policy, it is this way of thinking which appears to have had most influence on current approaches to incentivising research impact in the UK. As we shall see, however, the approach is widely criticised by the alternative theories of research-policy relations we explore later in the article.

A second popular theory of how research shapes policy emerges from Weiss’ ( 1977 , 1979 ) notion of the ‘enlightenment’ function of knowledge in policymaking. This account proposes that knowledge shapes policy through diffuse processes, resulting from the activities of various, overlapping networks, which contribute to broader, incremental and often largely conceptual changes (Hird, 2005 ; Walt, 1994 ). Radaelli’s ( 1995 ) notion of ‘knowledge creep’ is one of several more recent conceptualisations to build on this idea, and we can find similar assumptions in ideational theories of policy change (Béland, 2009 ; Hall, 1993 ; Schmidt, 2008 ). The implication of these accounts is that research influences policy over long periods through gradual changes in actors’ perceptions and ways of thinking (an idea that is also evident in theories of co-production, as discussed later) rather than through immediate, direct impacts. Whilst this body of work does not discount the possibility that research might contribute to what eventually become significant shifts in policy approaches, it suggests that assessments aiming to trace the impact of research on particular policy outcomes are likely to miss a potentially broader, more diffuse kind of conceptual influence.

The implications of this way of conceptualising the relationship between academic knowledge and policy for ideas about research impact are more challenging (indeed, the ‘enlightenment’ model has been criticised by some scholars seeking to improve the use of evidence in policy for its lack of practical utility (Nutley et al., 2007 )). Taking the more conceptual influence of research seriously suggests that incentives for achieving impact ought to shift away from individual researchers and projects to consider how to support the collective diffusion of much more diverse (potentially interdisciplinary) bodies of work. Given that multiple authors are likely to be involved, and that various factors unrelated to the underpinning research (or its communication) are likely to inform when and how knowledge shapes policy, it seems to make little sense to reward individual researchers (or even teams of researchers) for ‘achieving’ research impact. Instead, research impact might be supported by encouraging groups of researchers to work together on developing policy messages from diverse studies on particular policy topics (or, to support knowledge brokers to do this kind of work).

This is a very different model from both the RCUK pathways to impact approach, which encourages individual researchers or research teams to try to achieve research impacts on the back of single studies, and the REF impact case study approach, which encourages single institutions to narrate stories of impact based solely on the work of researchers they employ. Indeed, recent assessments of the REF impact case study approach have specifically highlighted the tendency not to adequately support these kinds of synthesised approaches to achieving impact (Manville et al., 2015 ; Smith and Stewart, 2016 ). For the moment, while some of the guidance documents relating to the UK impact agenda do acknowledge conceptual forms of influence, the mechanisms for monitoring and rewarding impact seem preoccupied with ‘instrumental’ research impact achieved on the back of research undertaken by individual researchers or small groups within single institutions.

Politics shapes knowledge

Perhaps the most obvious critique of the ‘knowledge shapes policy’ model reverses this relationship to highlight the various ways in which policies and politics shape knowledge and the use of knowledge. There is a rich body of literature theorising how state-building and modern techniques of governance have shaped the production of social knowledge (Foucault, 1991 , Heclo, 1974 ; Rueschemeyer and Skocpol, 1996 ), as well as how power relations are implicated in the construction of expert authority (Gramsci, 2009 ). What these diverse contributions share is the notion that an underlying political project is driving research production and utilisation, whether that project is the production of self-regulating subjects (as some Foucauldian interpretations suggest) or the continuing dominance of ruling elites and ideologies (as Gramscian analyses tend to posit). From this perspective, research utilisation in policymaking is understood as profoundly constrained; whilst those involved in the construction of policy are not necessarily consciously aware of the forces shaping their decisions, any attempt to engage with research must be understood as part of a wider political project. At the very least, such analyses suggest that only research that can be used to support these dominant ideas and interests will be employed in policymaking, while research that challenges dominant ideas will be discounted (see Wright et al., 2007 ). A stronger interpretation would hold that the research process is itself shaped by the ‘powerful interests’ directing policy agendas (e.g., Navarro, 2004 ).

The more applied literature concerning the relationship between research and policy also provides examples of this way of thinking about the relationship. In her overview of various ‘models’ of the relationship between research and policy, Weiss, for example, describes what she calls the ‘political model’, where research is deployed to support pre-given policy preferences; as well as a ‘tactical model’, where research is used as a method of delaying the decision-making process, providing policymakers with some ‘breathing space’ (Weiss, 1979 ). In the first case, the research process itself is not necessarily informed by politics but the decision to employ research (or not) is entirely political. In other words, political ideology and/or more strategic party politics inform the ways in which political actors respond to research evidence (e.g., Bambra, 2013 ). In the second, the commissioning of research might itself be understood as a political act (or, at least, an act that creates political benefits–see Bailey and Scott‐Jones 1984 ). In either case, efforts to reward researchers for ‘achieving’ research impact would seem misplaced.

The extent to which politics can shape research is perhaps most overt in research that is directly commissioned by sources with particular political/policy interests; reviews have repeatedly demonstrated that research funded by commercial sources, such as the pharmaceutical (e.g., Lundh et al., 2012 ) and tobacco industries (e.g., Bero, 2005 ), is more likely to present findings that are useful to those interests (see also Bailey and Scott‐Jones, 1984 ). In other contexts, it has been suggested that researchers may struggle to maintain their independence where research is commissioned directly, or indirectly, by government sources (e.g., Barnes, 1996 ; Smith, 2010 ). This kind of political influence may be felt both overtly and subtly, with researchers responding to signals from research funders as to what is likely to be funded (and what is not), what they are hoping (or expecting) to be found and what they are not (Knorr-Cetina, 1981 ; Smith, 2010 ), as we discuss further in the following section.

A second group of theories which call attention to the way in which politics can shape knowledge focus on the impact of institutions and organisational structures on policymaking and research. Similar to the previous group of theories, such accounts assume that the wider structures in which actors are located are key to explaining policy outcomes. Whilst the more political accounts discussed above highlight the ways in which power relations and elite interests can shape research and its use, these theories focus on organisational and decision-making structures. The most well-known of such theories are the various forms of institutionalism, of which ‘historical institutionalism’ is one of the most widely employed forms (see Immergut, 1998 for an overview). From this perspective, rather than constituting the collective result of individual preferences, policy processes (including efforts to engage with research) are considered to be significantly shaped by the historically constructed institutions and policy procedures within which they are embedded (Immergut, 1998 ).

Those who have contributed to the development of this genre of work have emphasised that such theories do not suggest that particular policy outcomes are inevitable –and indeed, as we discussed in the previous sections, under certain conditions existing paradigms can be superseded by new ideas, leading to substantial policy change (Hall, 1993 ). However, such theories do suggest that it becomes increasingly difficult to change the overall direction of a policy trajectory as previous decisions become ever more deeply embedded in institutional structures and ways of thinking (e.g., Kay, 2005 ). Employing these kinds of theories, Smith ( 2013b ) has demonstrated how the institutionalisation of particular ideas about health and economic policy function as filters to research-based ideas about health inequalities, encouraging those ideas that support existing institutionalised ideas (or ‘policy paradigms’) to move into policy, while blocking or significantly transforming more challenging ideas.

This way of thinking about the relationship between knowledge and policy suggests that research is constantly being influenced by policy and politics and that efforts to bring researchers and policymakers closer together are like to exacerbate this in ways that may not be desirable. At best, from this perspective, the research impact agenda seems likely to reward some academics (and not others) for achieving impacts that had far more to do with political interests and agendas than the research or impact activities of those academics. At worst, the impact agenda will lead to the increasing politicisation of research (and an associated reduction in academic freedom). Indeed, some of the most critical responses to the impact agenda are informed by these kinds of concerns. Cohen ( 2000 ) and Hammersley ( 2005 ), for example, have warned that the restrictions being placed on publicly-funded research to be ‘useful’ to policy audiences is limiting the potential for academics to promote ideas that are out-of-line with government policies. Likewise, Davey Smith et al., ( 2001 ), argue that efforts to achieve evidence-based policy may, in fact, do more to stimulate research that is shaped by policy needs than to encourage better use of research in policy-making.

Co-production

A third way of theorising research-policy relations has emerged from science and technology studies (STS), and posits a much more complex inter-relationship between knowledge production and governance. This approach is encapsulated in the idea of ‘co-production’: the claim that knowledge and governance are mutually constitutive (Jasanoff, 2004 ).

Similar to the approaches discussed in the last section, such accounts see knowledge as profoundly shaped by politics. But the notion of co-production focuses not just on the social and political constitution of science. It is also attentive to the other direction of influence: the ways in which governance is itself constituted by scientific knowledge. So rather than limiting its attention to how politics shapes knowledge, the notion of co-production posits that scientific and expert knowledge contribute to the construction of political reality (an idea that is, in some ways, simply a stronger version of Weiss’ ( 1979 ) account of the enlightenment function of research, discussed earlier). Knowledge provides the concepts, data and tools that underpin our knowledge of social and policy problems and appropriate modes of steering (Voß and Freeman, 2016 ). Sheila Jasanoff ( 2004 ) is arguably the most influential exponent of this approach. In her book States of Knowledge , she explores how knowledge-making is an inherent part of the practices of state-making and governance. States ‘are made of knowledge, just as knowledge is constituted by states’ (Jasanoff, 2004 , p 3). Moreover, STS scholars have shown how science does not just produce knowledge and theories that help define social problems and appropriate responses. It also produces skills, machines, instruments and technologies that are deployed in governance (Pickering, 1995 ).

An important concept informing this approach is that of performativity. This is the idea that social enquiry and its methods are ‘productive’: rather than simply describing social reality, they help to make or enact the social world (Law and Urry, 2004 ). Indeed, social science needs to be understood as fundamentally embedded in, produced by, but also productive of the social world (Giddens, 1990 ). Social science thus has effects–it creates concepts and labels, classifications and distinctions, comparisons and techniques that transform the social world. Such concepts and techniques can also help bring into existence the social objects they describe. Osborne and Rose ( 1999 ) illustrate this idea with the case of public opinion, a social phenomenon that was effectively created in the 1930s through the emergence of new methods of polling and survey analysis, and is now thoroughly normalised as an object of social scientific enquiry. Similarly, Donald MacKenzie ( 2006 ) has explored the performativity of economic models, showing how the theory of options shaped practices in trading and hedging in the financial sector from the 1970s onwards. Similar ideas have been explored by Colin Hay ( 2007 ) in his discussion of political disaffection. He argues that public choice theory has contributed to the ‘marketisation’ of party politics, implying that such theories have been ‘performative’ (although he does not use this term).

Theories of co-production also show how science can produce social problems. Through its various scientific and technical innovations, science does not simply solve governance problems, but it also creates new ones (Jasanoff, 2004 ). The frantic pace of development and progress in science and technology produce a continuous stream of new problems and solutions, which governments often struggle to keep pace with. So new research does not just offer ways of ordering the social world, but can also destabilise existing structures and modes of governance. In areas of policy that are highly dependent on technology and science–such as energy, health, agriculture or defence - policy develops almost in pursuit of science, in an attempt to catch up with, harness and regulate the new technologies and practices it has produced. Thus science creates the very problems that need to be addressed through political intervention (Beck, 1992 ). The demand for ever more problem-solving knowledge is effectively built into the structure of policy-research relations.

What implications do these approaches have for defining and measuring impact? First, they suggest that we cannot neatly disentangle processes of knowledge production from those of governance. This is not merely an epistemological question–a challenge of finding the right methods or observational techniques to allow us to separate out how social scientific findings have influenced politics or policy (although this is of course difficult to do). It represents a more fundamental ontological problem, in that social scientific knowledge is co-constitutive of politics. Imagine, for example, trying to chart the ‘impact’ of public choice theories on politics. We would not only face the methodological challenge of charting the subtle and incremental processes through which a wide variety of social actors (including politicians, campaigners, lobbyists and the media) appropriated public choice theories about political agency. We would also need to understand the ongoing feedback effects through which such ideas brought about shifts in the behaviour of these actors, in turn gradually transforming political behaviour. If we accept the possibility of such effects, then we need to also consider how such shifts may in turn validate the theories that originally produced them, enhancing their authority and influence. The relationship between social science and politics in this example is one of continuous mutual influence and reinforcement.

Second, the notion of co-production suggests that social science may itself produce social problems that require political responses. Studies of public opinion offer a good example of this. A survey of public attitudes may ‘discover’ unarticulated claims and preferences, which produce new demands for political action. In 2014, Jeffery et al., ( 2014 ) found a strong desire on the part of the English respondents they surveyed for institutions that better represented and articulated ‘English’ views. This could be charted as ‘impact’ insofar as the findings of the survey were picked up by politicians and influenced claims-making about UK constitutional reform (and indeed it was submitted as a case study to REF2014). But the research can also be understood as producing a new set of political problems. It encouraged a number of survey respondents to articulate a set of preferences which may previously have been nascent or unspecified. These preferences were then presented as a collective and coherent political claim, which in turn implied the need for enhanced political representation and constitutional reform. Research thus contributed to the construction of a new social problem requiring a political response. As with the case of public choice theory, we can also posit a feedback effect, whereby the social and political adjustments generated by the research might in turn further validate the findings. As politicians sought to represent and mobilise these preferences, this created further political expectations and demands, thereby substantiating the initial research claim that the English desire their own institutions.

One implication of this account is that REF or HEFCE models do not do justice to the more pervasive (but often subtle) influence of social science on policy. Another is that they overlook the feedback effects described above, whereby the political adjustments enacted through social science in turn validate (or possible discredit) the authority of research findings or methods. And a third is that they may actively encourage forms of interference that create more problems than they solve. Policy impact may not always be benign, as we noted earlier.

Assuming we accept such impacts as desirable, how might these processes of co-production be best captured and accredited? They would require quite resource-intensive methodologies, as well as forms of expertise that are not necessarily available across disciplines. Each case study would effectively be a social scientific project in its own right, explored though a range of qualitative and quantitative methods, such as ethnography (as Baim-Lance and Vindrola-Padros, 2015 , argue in more detail) process tracing, discourse analysis, interviews and surveys. It is hard to imagine sufficient resource being available for such indepth enquiry, or, indeed, for buy-in to such models and methodologies from across (non-social science) disciplines.

Autonomous spheres

Our final approach to theorising research-policy relations understands science and politics as distinct spheres, each operating according to a separate logic and system of meaning. As we saw earlier, one version of this account is Caplan’s ( 1979 ) ‘two communities’ thesis, which identifies a ‘cultural gap’ between researchers and policymakers. This conceptualisation has been subject to a range of critiques, not least, as Lindquist ( 1990 ) points out, the fact that this way of thinking about the relationship excludes a range of potentially important actors, such as journalists, consultants and lobbyists. Despite this, whilst not always referring to Caplan’s ( 1979 ) work directly, many contemporary assessments of the limited use of research in policy and practice frequently mirror Caplan’s observations by highlighting perceived ‘gaps’ between researchers, policymakers and/or practitioners as a fundamental barrier to the use of research.

In this section, we focus on a more radical account of this ‘gap’, associated with the systems theory of German sociologist Niklas Luhmann (e.g., Luhmann, 1996 ). On a Luhmannian systems theory account, science and politics are both understood as self-referential or ‘autopoietic’ systems. Although mutually dependent in important ways (they could not survive in a recognisable form without one another), each operates according to its own logic or ‘communicative code’, which determines which communications are relevant to the system. There is no causality or direct influence across systems: rather, operations in one system are selectively perceived and given meaning according to the codes and logics of another system. Thus it does not make sense to conceive of flows, diffusion or causality across systems, and STS concepts such as ‘performativity’ or ‘co-production’ need to be carefully re-specified in terms of how one system ‘models’ and responds to the operations of another.

Luhmann understands the primary building blocks of modern society not as individuals or groups, but as functionally differentiated social systems. Modern societies are increasingly sub-divided into specialised, self-referential systems such as education, health, economy, religion, welfare, science or politics. Each of these systems operates according to its own distinct codes, programmes, logic and mode of inclusion. Unlike on Caplan’s account, these systems are not distinguished in terms of members or institutions. Systems do not consist of discrete groups of people, indeed one person or one organisation can participate in several different systems. However, systems are distinguished in terms of sets of differentiated roles and activities. Each system retains its distinctiveness through developing its own criteria of selection, which help it reduce complexity by only selecting those communications which are relevant to the system.

On this account, science and politics are separate function-systems. Science (including social science) operates according to a binary code of true/false. In other words, it defines relevant communication based on whether it is concerned with establishing truth claims. The system of politics, meanwhile, selects relevant communication on the basis of the binary code of government/opposition. The political system selects and gives meaning to communication based on its relevance to the pursuit of political power and the capacity to adopt collectively binding decisions. At first sight, this seems to be a very narrow way of conceiving social systems. For example, scientists are not just preoccupied with validating truth claims; they are clearly also concerned with winning grants, enhancing their academic reputation, or influencing government policies. But these preoccupations are characterised as participating in different systems. For example, a public funding decision has a distinct meaning and relevance in the systems of science, politics and the economy.

From this perspective, there can be no overarching causality operating between two systems, although it is easy to see how appealing such causal attributions might be to observers. To be sure, one event can have effects across different systems. A government research grant has meaning for both the system of politics and that of science. Yet As Luhmann puts it, the ‘preconditions and consequences of events differ completely according to system reference’, and observers should not ‘cross-identify events over boundaries’ (Luhmann, 1991 , p 1438). Instead, Luhmann conceives of the relationship as highly selective connections between systems and their environments. Systems that are reliant on other systems in their environment develop models, or assumed regularities, to help them keep tabs on the other system. For example, science will develop a certain way of observing and anticipating political decision-making relevant to science: a set of beliefs about how and when decisions are produced, what drives them, and what effects they may have on science funding or regulation. These models can be understood as internally constructed filters to help select what is relevant from what is noise or redundancy. They help the system to sort through what is expected and what is unpredicted, what is a relevant signal and what is an irritation (Luhmann, 1991 , p 1432).

If we accept that science and politics are guided by distinct logics or communicative codes, the challenge becomes one of reconstructing how each system might selectively pick up signals from the other. We need to understand what sort of perceptual filters are developed and stabilised for the purpose of screening out relevant signals from noise; and how information from the other system might be constructed and connected to the receiving system’s identities and functions. The implication is that we need to turn our attention to how the system of politics ‘models’ the system of science, and how it selectively appropriates and gives meaning to the signals produced by that system.

This segues nicely into the earlier discussion of our first set of theories, and the need for a more sophisticated theory of politics than those provided by prevalent models of research-policy relations. Such a theory would require an account of how the political system makes sense of its environment, and selectively draws on different types of resources to secure legitimacy or support (Boswell, 2009a ). A number of theories from public policy can contribute towards such an endeavour. Notably, theories of information-processing offer potential to examine how organisations in the public administration selectively pick up signals from their environment about social problems (e.g., Baumgartner and Jones, 1993 ). Cohen and colleagues’ ( 1972 ) ‘garbage can model’ of policymaking, as taken up by Kingdon ( 1995 [1984]), offers a neat way of theorising how different ideas or ‘solutions’ are picked up depending on the political and problem streams–again, an idea broadly compatible with the systems theory approach, in that it views ‘ideas’ and ‘politics’ as operating according to different temporalities and logics (Boswell and Rodrigues, 2016 ).

What are the implications of systems theory for impact? A systems theoretic approach would be wary of the attempt to demonstrate ‘impact’, as it assumes a specious causality between science and politics. Instead, we need to try to adopt the perspective of politics, and make sense of how and why the political system picks up data, methods or techniques from social science. And we can attempt to observe how, from the perspective of social science, political decisions or goals might affect the selection and framing of research questions, and the communication of research findings. But we cannot integrate these observations into a single set of causal mechanisms. Viewed from ‘inside’ of each system, the other remains a ‘black box’: an infinitely complex set of communications and operations which can only be very crudely modelled and selectively responded to.

What this implies is that an impact case study could at best chart how politics appropriated and gave meaning to particular data, methods or techniques. But the ‘underpinning research’ that produced these data or techniques, or academic efforts to promote this research, would derive rather limited credit for such take-up. Far more important would be dynamics internal to the political system, such as the political salience of the issue, or how well the research in question was attuned to dominant political framings of policy problems (Kingdon, 1995 [1984]; Cairney, 2016 ), or how far research was seen as an authoritative mode of knowledge for guiding decisions (Boswell, 2009b ). Moreover, it would remain open how far political take-up reflected a preoccupation with signalling legitimacy, rather than informing policy interventions. After all, if research is valued by politics as a means of substantiating claims or bolstering credibility, then presumably this implies a symbolic rather than instrumental rationale for using research (Boswell, 2009a ).

In short, the systems theoretic account guides us towards an interrogation of the political context of knowledge utilisation; but the more we probe the logic of knowledge appropriation in politics, the less we can accredit research. What makes for politically useful knowledge is fundamentally distinct from what makes for good science. Thus any link between high quality science and impact is exposed as contingent. It may well be that politics needs to ‘quality control’ the science it invokes to insure against its invalidation by critics–but this is only as an insurance against critique. And it may want to ensure the robustness of science as a safeguard against making mistakes that would cost political support. But again, this concern with rigour is incidental to the core concerns of politics. Politics is not fundamentally preoccupied with what is true, but with what is relevant to securing power and producing collectively binding decisions.

Current approaches to research impact appear to have been informed by simplistic supply-side models within our first category of ‘knowledge shapes policy’. As we have suggested in this article, such accounts have been widely debunked by theorists of research-policy relations, as well as by many empirical studies of research ‘impact’. And yet the REF and HEFCE models, and much of the literature on knowledge utilisation, continue to remain faithful to this problematic account. Part of the reason for the sustained commitment to these models is that they offer a reassuring narrative to both policy-makers and researchers. Politicians and public servants can demonstrate the rigour and authority of their claims by invoking research, and they can secure legitimacy by signalling that their decisions are well-grounded (Boswell, 2009a ), or they can invoke the need for research as a rationale for delaying action (Fuller, 2005 ). At the same time, researchers can secure additional resources and credit for developing compelling narratives about the impact of their research (Dunlop, 2017 ). Yet these accounts bely the complexity of research-policy relations and, indeed, of policy processes and policy change (Cohen et al., 1972 ; Smith and Katikireddi, 2013 ). If we are to avoid continually reinventing broken wheels, we suggest a new, more theoretically informed approach to thinking about research impact is required.

The existing literature on research impact has already subjected current approaches to assessing, incentivizing and rewarding impact in the UK to extensive critique, and it was not the purpose of this paper to expand on these critiques. Rather, our aim has been to set out four alternative, sophisticated accounts of the relationship between research and policy and to consider what a research impact agenda might look like if it were informed by these other approaches. Such an exercise is necessarily hypothetical and almost impossible to test in an empirical sense, since the UK’s approach to research impact has already been informed by a relatively simple and linear conceptualisation of research-policy relations (Smith and Stewart, 2016 ). This means there are strong incentives for institutions to ‘play the game’ according to the rules that have been set by providing relatively simple and linear ‘stories’ of research impact, as Meagher and Martin’s ( 2017 ) analysis of REF2014 impact case studies for mathematics attests (see also Murphy, 2017 on ‘gaming’ in REF and Watermeyer and Hendgecoe 2016 on ‘impact mercantilsm’). However, as other countries evolve different approaches to research impact, it may become possible to empirically assess both the claims we set out here and the practical implications of such alternative approaches.

The first of the four models we outline offered a subtler ‘enlightenment’ conception of how research can influence policy. It implied that research can lead to ideational adjustments through diffuse and incremental processes, typically influenced by a wide body of research rather than individual findings. This account challenges the notion that researchers or institutions should be rewarded for claims about the impact of individual studies, though potentially supports efforts to encourage knowledge exchange. The second set of theories implied that policy and politics shape knowledge production and use, and were altogether more sceptical of the impact agenda. They suggested that it was naïve to assume that researchers can speak truth to power, implying that researchers should not be rewarded for their supposed impact since policy actors employ research for political, rather than empirical/intellectual, reasons. The third set of theories on co-production implied the need for a far more sophisticated methodology for examining how research and governance are mutually constitutive. They also argued that social science should not necessarily be understood as the ‘solution’ to social problems, since it can itself create such problems. And the fourth approach, which posits that science and politics are autonomous systems, suggested that we can best understand impact through a theory of how politics selectively observes and gives meaning to communications emanating from the system of science. Viewed from this perspective, the impact agenda has been designed to suit the needs of a political, rather than scientific, system and should be treated cautiously by researchers given its potential to divert science from its core task of developing truth claims.

Both the second and fourth accounts suggest that the very idea of trying to incentivize the use of research in policy is flawed. On these accounts, we should be cautious about adopting systems that reward researchers for influencing policy. Such impacts are spurious, in that their apparent influence is down to pre-given interests or independent political dynamics; or they are the result of researchers aligning research questions and approaches to pre-fit political agendas. By rewarding researchers for achieving impact we are adopting an arbitrary incentive system that is at best decoupled from research quality, and at worst, threatens the integrity and independence of social science.

For those more sympathetic to the idea of ‘research impact’, the first and third approaches might offer more hope. Nonetheless, neither approach suggests that the current approach is likely to achieve its intended goals. Indeed, both caution against rewarding individual researchers for ‘achieving’ research impact based on narrow indicators (e.g., citations in policy documents). The enlightenment model suggests that research impact involves subtle, incremental and diffuse ideational adjustments over a long period of time, which are generated by a wide range of research insights rather than specific individual findings. This suggests that a system for rewarding impact should not focus on individual research projects or groups and their linear effects on particular policies. Rather, impact frameworks should reward collaborative endeavours that build incrementally on a wider body of work; that develop longer-term relationships with a range of non-academic audiences (not only policymakers and other ‘elites’); and that may bring about subtle conceptual shifts, rather than clearly identifiable policy changes. This in turn implies the need for more complex research designs and methodologies for charting such influence over a far longer time-frame, and avoid incentives to over-claim credit for particular groups or projects. This perspective coheres with those arguing for a shift away from trying to measure and incentivize research impacts to focus instead on incentivizing and rewarding knowledge exchange processes (e.g., Upton et al., 2014 ). From this view, Spaapen and van Drooge’s ( 2011 ) approach of focusing on ‘productive interactions’ between science and society (which emerged out of an FP7 project called Social Impact Assessment Methods for research and funding instruments-SIAMPI), seems like a more defensible means of assessing research impact. The notion of co-production similarly suggests the need for more in-depth, ethnographic or process-tracing methods for reconstructing the complex relationships between research and policy (as outlined by Baim-Lance and Vindrola-Padros, 2015 ). Systems for rewarding impact should also be aware of the two-way relationship between research and governance, including the ways in which social science can itself affect the social and political world, imagining and enacting new social problems.

Arguably, the highest impact research is that which serves to re-shape the social world it seeks to describe. This implies that models to promote engagement with knowledge users need to be attentive not just to the complex pathways to research impact, but also to the very real ethical implications of research influence (implications that do not currently appear to be considered in either REF impact case studies or RCUK pathways to impact statements–Smith and Stewart, 2016 ). Not only can the impact agenda affect the practices of social science, as is widely recognised in social science literature; social science can also instigate new policy problems. Proponents of policy impact should have a care what they wish for.

Data availability

The article does not generate or make use of any datasets.

Change history

20 february 2018.

On page 7 of the PDF, in the second paragraph under the subheading “Conclusion” the first sentence “The existing literature on research impact has already subjected current approaches to assessing, incentivizing and rewarding impact in the UK to extensive critique (e.g., ADD REFS) and it was not the purpose of this paper to expand on these critiques” has been corrected to “The existing literature on research impact has already subjected current approaches to assessing, incentivizing and rewarding impact in the UK to extensive critique, and it was not the purpose of this paper to expand on these critiques”.

AHRC (2014) What we do-strengthen research impact. http://www.ahrc.ac.uk/What-We-Do/Strengthen-research-impact/Pages/Strengthen-Research-Impact.aspx . Accessed 10 Jan 2014

AHRC (2015) Knowledge exchange and partnerships. http://www.ahrc.ac.uk/innovation/knowledgeexchange/ . Accessed 14 Oct 2015

Bailey LF, Scott‐Jones G (1984) Rational, irrational and other reasons for commissioning research. Mark Intell Plan 2(3):36–50

Article   Google Scholar  

Baim-Lance A, Vindrola-Padros C (2015) Reconceptualising ‘impact’ through anthropology’s ethnographic practices. Anthropol Action 22(2):5–13

Bambra C (2013) The primacy of politics: the rise and fall of evidence-based public health policy? J Public Health 35:486–487

Barnes C (1996) Disability and the myth of the independent researcher. Disabil Soc 11:107–112

Baumgartner F, Jones B (1993) Agendas and instability in American politics. University of Chicago Press, Chicago

Google Scholar  

Beck U (1992) The risk society: towards a new modernity, Sage

Béland D (2009) Ideas, institutions, and policy change. J Eur Public Policy 16:701–718

Bero LA (2005) Tobacco industry manipulation of research. Public Health Rep 120:200–208

Article   PubMed   PubMed Central   Google Scholar  

Blume S (1977) Policy as theory: a framework for understanding the contribution of social science to welfare policy. Acids Sociol 20:47–62

Blunkett D (2000) Influence or irrelevance: can social research improve government? Res Intell, BERA, 71:12–21

Bornmann L (2013) What is societal impact of research and how can it be assessed? a literature survey. J Assoc Inf Sci Technol 64(2):217–233

Boswell C (2009a) The political uses of expert knowledge-immigration policy and social research. Cambridge University Press, Cambridge, UK

Book   Google Scholar  

Boswell C (2009b) Knowledge, legitimation and the politics of risk: the functions of research in public debates on migration. Polit Stud 57:165–186

Boswell C, Rodrigues E (2016) Policies, politics and organizational problems: multiple streams and the implementation of targets in UK government. Policy Polit 44(4):507–524

Brewer J (2011) Viewpoint—from public impact to public value. Methodol Innov Online 6:9–12

Cabinet Office (1999) Modernising government (White Paper). The Stationary Office, London

Cabinet Office (2000) Wiring it up: whitehall’s management of cross-cutting policies and services: a performance and innovation unit report. Cabinet Office, London

Canadian Academy of Health Sciences (CAHS) (2009) Making an impact: a preferred framework and indicators to measure returns on investment in health research. CAHS, Ottawa, Ontario

Cairney P (2016) The politics of evidence-based policy making. Springer

Caplan N (1979) The two-communities theory and knowledge utilization. Am Behav Sci 22:459–470

Chubb J, Watermeyer R (2016) Artifice or integrity in the marketization of research impact? Investigating the moral economy of (pathways to) impact statements within research funding proposals in the UK and Australia, Studies in Higher Education Advance Online Access: https://doi.org/10.1080/03075079

Clarke S (2010) Pure science with a practical aim: the meanings of fundamental research in Britain, circa 1916–1950. Isis 101:285–311

Article   PubMed   Google Scholar  

Cohen MD, March JG, Olsen JP (1972) A garbage can model of organizational choice. Adm Sci Q 17:1–25

Cohen N (2000) With our money, they hide the truth. The New Statesman, 20(3):17–19

Contandriopoulos D, Lemire M, Denis J-L, Tremblay É (2010) Knowledge exchange processes in organizations and policy arenas: a narrative systematic review of the literature. Milbank Q 88:444–483

Davey Smith G, Ebrahim S, Frankel S (2001) How policy informs the evidence. BMJ 322:184–185

Article   CAS   PubMed   PubMed Central   Google Scholar  

Donovan C (2008) The Australian research quality framework: a live experiment in capturing the social, economic, environmental and cultural returns of publicly funded research. New Dir Eval 118:47–60

Donovan C, Hanney S (2011) The ‘Payback framework’ explained. Res Eval 20(3):181–183

Dunlop C (2017) Narrating impact. The politics of autopoietic narratives. Paper presented at the annual political studies association conference, University of Strathclyde, March 2017

ESRC (2014a) How to maximize research impact. www.esrc.ac.uk/funding-and-guidance/toolsand-resources/how-to-maximise-impact/ . Accessed 10 Jan 2014

ESRC (2014b) Impact toolkit. http://www.esrc.ac.uk/funding-and-guidance/tools-andresources/impact-toolkit/ . Accessed 10 Jan 2014

ESRC (2014c) What is research impact? www.esrc.ac.uk/research/evaluation-and-impact/what-is-research-impact/ Accessed 10 Jan 2014.

Foucault M (1991) Discipline and punish: the birth of the prison (penguin social sciences)

Fuller S (2005) The intellectual. Icon Books, Cambridge

Giddens A (1990) The consequences of modernity. Stanford University Press, Stanford

Gramsci (2009) Hegemony, intellectuals and the state. Cult Theory Pop Cult 2:210–16

Grant J, Brutscher P-B, Kirk SE, Butler L, Wooding S (2010) Capturing research impacts a review of international practice prepared for the higher education funding council for England. RAND Europe, Cambridge

Greenhalgh T, Raftery J, Hanney S, Glover M (2016) Research impact: a narrative review. BMC Medicine 201614:78 https://doi.org/10.1186/s12916-016-0620-8

Hall PA (1993) Policy paradigms, social learning, and the state: the case of economic policymaking in Britain. Comp Polit 25(3):275–296

Hammersley M (2005) Is the evidence-based practice movement doing more good than harm? Reflections on Iain Chalmers’ case for research-based policy making and practice. Evid Policy 1:85–100

Hay C (2007) Why we hate politics. Polity Press, Cambridge

Heclo H (1974) Modern social politics in Britain and Sweden: from relief to income maintenance. Yale University Press

Hird JA (2005) Policy analysis for what? the effectiveness of nonpartisan policy research organizations. Policy Stud J 33:83–105

Immergut EM (1998) The theoretical core of the new institutionalism. Polit Soc 26(1):5–34

Innvaer S, Vist G, Trommald M, Oxman A (2002) Health policymakers’ perceptions of their use of evidence: a systematic review. J Health Serv Res Policy 7:239–244

Jasanoff S (ed) (2004) States of knowledge: the co-production of science and the social order. Routledge, London

Jeffery C, Wyn Jones R, Henderson A, Scully R, Lodge G (2014) Taking England seriously: the new english politics - the future of England survey 2014. Centre on Constitutional Change, University of Edinburgh, Edinburgh

Katikireddi SV, Higgins M, Bond L, Bonell C, Macintyre S (2011) How evidence based is English public health policy? BMJ 343

Kay A (2005) A critique of the use of path dependency in policy studies. Public Administration 83(3):553–571

Kingdon JW (1995)[1984] Agendas, alternatives, and public policies. 2nd edn. HarperCollins College Publishers, New York, NY

Knorr-Cetina KD (1981) The manufacture of knowledge: an essay on the constructivist and contextual nature of science. Pergamon

Labour Party (1997) New labour because britain deserves better: the labour party manifesto. Labour Party, London

Lavis JN (2006) Research, public policymaking, and knowledge-translation processes: Canadian efforts to build bridges. J Contin Educ Health Prof 26:37–45

Law J, Urry J (2004) Enacting the social. Econ Soc 33:390–410

Lindquist EA (1990) The third community, policy inquiry, and social scientists. In: Brooks S, Gagnon A-G (eds) Social scientists, policy, and the state. Praeger, New York, NY, p 21–51

Lomas J (2000) Using ‘linkage and exchange’ to move research into policy at a Canadian foundation. Health Aff 19:236–240

Article   CAS   Google Scholar  

Luhmann N (1991) Operational closure and structural coupling: the differentiation of the legal system. Cardozo Law Rev 13:1419–1441

Luhmann N (1996) Social systems. Stanford University Press, Stanford

Lundh A, Sismondo S, Lexchin J, Busuioc OA, Bero L (2012) Industry sponsorship and research outcome (Review). Cochrane Database of Systematic Reviews: Cochrane Library

MacKenzie D (ed) (2006) An engine, not a camera. How financial models shape the markets. MIT Press, Cambridge, MA

Manville C, Guthrie S, Henham ML, Garrod B, Sousa S, Kirtley A et al. (2015) Assessing impact submissions for REF2014: an evaluation. https://www.rand.org/pubs/research_reports/RR1032.html . Accessed 11 May 2017: RAND

Meagher LR, Martin U (2017) Slightly dirty maths: the richly textured mechanisms of impact. Res Eval 26(1):15–27

Mitton C, Adair CE, McKenzie E, Patten SB, Waye Perry B (2007) Knowledge transfer and exchange: review and synthesis of the literature. Milbank Q 85:729–768

Morton S (2015) Progressing research impact assessment: a ‘contributions’ approach. Res Eval 24(4):405–419

Article   MathSciNet   Google Scholar  

Mostert S, Ellenbroek S, Meijer I, van Ark G, Klasen E (2010) Societal output and use of research performed by health research groups. Health Res Policy Syst 8(1):30

iMRC (2014) Achievements and impact. www.mrc.ac.uk/achievementsandimpact/ Accessed 10 Jan 2014

Murphy T (2017) Revising the research excellence framework: ensuring quality in REF2021 or new challenges ahead? Perspect 21(1):34–39

Naughton M (2005) ‘Evidence-based policy’ and the government of the criminal justice system-only if the evidence fits! Crit Social Policy 25:47–69

Navarro V (2004) The politics of health inequalities research in the united states. Int J Health Serv 34(1):87–99

Nutley S, Davies HTO, Walter IC (2003) Evidence-based policy & practice: cross-sector lessons form the United Kingdom. Soc Policy J NZ, pp 29–48

Nutley S, Walter I, Davies H (2007) Using evidence: how research can inform public services. Policy Press, Bristol

Oliver K, de Vocht F, Money A, Everett MG (2013) Who runs public health? A romixed-methods study combining network and qualitative analyses. J Public Health 35:453–459

Osborne T, Rose N (1999) Do the social sciences create phenomena?: the example of public opinion research. Br J Sociol 50:367–396

Article   CAS   PubMed   Google Scholar  

Parsons W (2002) From muddling through to muddling up-evidence based policy making and the modernisation of British government. Public Policy Adm 17:43–60

Pickering A (1995) The mangle of practice: time, agency, and science. The University of Chicago Press, Chicago

Book   MATH   Google Scholar  

Radaelli CM (1995) The role of knowledge in the policy process. J Eur Public Policy 2:159–183

REF 2014 (2011) Assessment framework and guidance on submissions. http://www.ref.ac.uk/media/ref/content/pub/assessmentframeworkandguidanceonsubmissions/GOS%20including%20addendum.pdf . Accessed 15 May 2017

REF 2014 (2014) Assessment criteria and level definitions. http://www.ref.ac.uk/panels/assessmentcriteriaandleveldefinitions/ . Accessed 11 May 2017

Rein M (1980) Methodology for the study of the interplay between social science and social policy. Int Soc Sci J 22:361–368

Research Councils UK (Undated) RCUK Review of Pathways to Impact: Summary. http://www.rcuk.ac.uk/documents/documents/ptoiexecsummary-pdf/ . Accessed 11 May 2017

Rueschemeyer D, Skocpol T (eds) (1996) States, social knowledge, and the origins of modern social policies. Princeton University Press

Schmidt VA (2008) Discursive institutionalism: the explanatory power of ideas and discourse. Annu Rev Polit Sci 11:303–326

Smith K (2010) Research, policy and funding–academic treadmills and the squeeze on intellectual spaces1. Br J Sociol 61:176–195

Smith KE (2013a) Beyond evidence-based policy in public health: the interplay of ideas. Palgrave Macmillan, Basingstoke

Smith KE (2013b) Institutional filters: the translation and re-circulation of ideas about health inequalities within policy. Policy Polit 41(1):81–100

Smith KE, Katikireddi SV (2013) A glossary of theories for understanding policymaking. J Epidemiol Community Health 67(2):198–202

Smith KE, Stewart E (2016) We need to talk about impact: why social policy academics need to engage with the UK’s research impact agenda. J Soc Policy 46:109–127

Spaapen J, van Drooge L (2011) Introducing ‘productive interactions’ in social impact assessment. Res Eval 20(3):211–21

Stevens A (2007) Survival of the ideas that fit: an evolutionary analogy for the use of evidence in policy. Social Policy Soc 6:25–35

UK Higher Education Funding Bodies (2011) Decisions on assessing research impact (REF 01.11). http://www.ref.ac.uk/media/ref/content/pub/decisionsonassessingresearchimpact/01_11.pdf . Accessed 11 May 2017

Upton S, Vallance P, Goddard J (2014) From outcomes to process: evidence for a new approach to research impact assessment. Res Eval 23(4):352–365

Voß J-P, Freeman R (eds) (2016) Knowing governance: the epistemic construction of political order. Palgrave Macmillan, Basingstoke

Walt G (1994) How far does research influence policy? Eur J Public Health 4:233–235

Walter I, Nutley S, Davis H (2005) What works to promote evidence-based practice? Evid Policy 1:335–363

Watermeyer R, Hendgecoe A (2016) Selling impact: peer-reviewer projections of what is needed and what counts in REF impact case studies. a retrospective analysis. J Educ Policy 31(5):651–665

Weiss C (1977) Research for policy’s sake: the enlightenment function of social research. Policy Anal 3:531–547

Weiss C (1979) The many meanings of research utilization. Public Adm Rev 39:426–431

Wright JSF, Parry J, Mathers J (2007) What to do about political context? Evidence synthesis, the New Deal for Communities and the possibilities for evidence-based policy. Evid Policy 3(2):253–269

Download references

Author information

Authors and affiliations.

University of Edinburgh, Edinburgh, UK

Christina Boswell & Katherine Smith

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Christina Boswell .

Ethics declarations

Competing interests.

The authors declare no competing financial interests.

Additional information

Publisher’s note : Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Change History A correction to this article has been published and is linked from the HTML version of this article.

A correction to this article is available online at https://doi.org/10.1057/s41599-018-0073-0 .

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Boswell, C., Smith, K. Rethinking policy ‘impact’: four models of research-policy relations. Palgrave Commun 3 , 44 (2017). https://doi.org/10.1057/s41599-017-0042-z

Download citation

Received : 24 August 2017

Accepted : 09 November 2017

Published : 12 December 2017

DOI : https://doi.org/10.1057/s41599-017-0042-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Combining public health evidence, policy experience and communications expertise to inform preventive health: reflections on a novel method of knowledge synthesis.

• Maddie Heenan
• Alexandra Chung
• Lucie Rychetnik

Health Research Policy and Systems (2023)

Advancing health through evidence assisted decisions with health policy and systems research program: a qualitative evaluation of a national health research grant management process in the Philippines

• Reneepearl Kim Sales
• Gladys Kaye Reyes-Ramos
• Maria Eufemia Yap

Evidence synthesis to policy: development and implementation of an impact-oriented approach from the Eastern Mediterranean Region

• Fadi El-Jardali
• Racha Fadlallah
• Elie A. Akl

Predicting factors of public awareness and perception about the quality, safety of drinking water, and pollution incidents

• Tarek Benameur
• Nassima Benameur
• Abdelali Agouni

Environmental Monitoring and Assessment (2022)

Mediating Knowledge Co-Production for Inclusive Governance and Delivery of Food, Water and Energy Services in African Cities

• Temilade Sesan
• Safietou Sanfo
• Rabani Adamou

Urban Forum (2022)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

• Search Menu

Sign in through your institution

• Advance articles
• Themed Collections
• Editor's Choice
• Ilona Kickbusch Award
• Supplements
• Author Guidelines
• Submission Online
• Open Access Option
• Self-Archiving Policy
• About Health Promotion International
• Editorial Board
• Advertising and Corporate Services
• Journals on Oxford Academic
• Books on Oxford Academic

• < Previous

A guide to policy analysis as a research method

• Article contents
• Figures & tables
• Supplementary Data

Jennifer Browne, Brian Coffey, Kay Cook, Sarah Meiklejohn, Claire Palermo, A guide to policy analysis as a research method, Health Promotion International , Volume 34, Issue 5, October 2019, Pages 1032–1044, https://doi.org/10.1093/heapro/day052

• Permissions Icon Permissions

Policy analysis provides a way for understanding how and why governments enact certain policies, and their effects. Public health policy research is limited and lacks theoretical underpinnings. This article aims to describe and critique different approaches to policy analysis thus providing direction for undertaking policy analysis in the field of health promotion. Through the use of an illustrative example in nutrition it aims to illustrate the different approaches. Three broad orientations to policy analysis are outlined: (i) Traditional approaches aim to identify the ‘best’ solution, through undertaking objective analyses of possible solutions. (ii) Mainstream approaches focus on the interaction of policy actors in policymaking. (iii) Interpretive approaches examine the framing and representation of problems and how policies reflect the social construction of ‘problems’. Policy analysis may assist understanding of how and why policies to improve nutrition are enacted (or rejected) and may inform practitioners in their advocacy. As such, policy analysis provides researchers with a powerful tool to understand the use of research evidence in policymaking and generate a heightened understanding of the values, interests and political contexts underpinning policy decisions. Such methods may enable more effective advocacy for policies that can lead to improvements in health.

Personal account

• Sign in with email/username & password
• Get email alerts
• Save searches
• Purchase content
• Activate your purchase/trial code
• Add your ORCID iD

Institutional access

Sign in with a library card.

• Sign in with username/password
• Recommend to your librarian
• Institutional account management
• Get help with access

Access to content on Oxford Academic is often provided through institutional subscriptions and purchases. If you are a member of an institution with an active account, you may be able to access content in one of the following ways:

IP based access

Typically, access is provided across an institutional network to a range of IP addresses. This authentication occurs automatically, and it is not possible to sign out of an IP authenticated account.

Choose this option to get remote access when outside your institution. Shibboleth/Open Athens technology is used to provide single sign-on between your institution’s website and Oxford Academic.

• Click Sign in through your institution.
• Select your institution from the list provided, which will take you to your institution's website to sign in.
• When on the institution site, please use the credentials provided by your institution. Do not use an Oxford Academic personal account.
• Following successful sign in, you will be returned to Oxford Academic.

If your institution is not listed or you cannot sign in to your institution’s website, please contact your librarian or administrator.

Enter your library card number to sign in. If you cannot sign in, please contact your librarian.

Society Members

Society member access to a journal is achieved in one of the following ways:

Sign in through society site

Many societies offer single sign-on between the society website and Oxford Academic. If you see ‘Sign in through society site’ in the sign in pane within a journal:

• Click Sign in through society site.
• When on the society site, please use the credentials provided by that society. Do not use an Oxford Academic personal account.

If you do not have a society account or have forgotten your username or password, please contact your society.

Sign in using a personal account

Some societies use Oxford Academic personal accounts to provide access to their members. See below.

A personal account can be used to get email alerts, save searches, purchase content, and activate subscriptions.

Some societies use Oxford Academic personal accounts to provide access to their members.

Viewing your signed in accounts

Click the account icon in the top right to:

• View your signed in personal account and access account management features.
• View the institutional accounts that are providing access.

Signed in but can't access content

Oxford Academic is home to a wide variety of products. The institutional subscription may not cover the content that you are trying to access. If you believe you should have access to that content, please contact your librarian.

For librarians and administrators, your personal account also provides access to institutional account management. Here you will find options to view and activate subscriptions, manage institutional settings and access options, access usage statistics, and more.

Short-term Access

To purchase short-term access, please sign in to your personal account above.

Don't already have a personal account? Register

Email alerts

Citing articles via.

• Recommend to Your Librarian
• Journals Career Network

Affiliations

• Online ISSN 1460-2245
• Print ISSN 0957-4824
• Copyright © 2024 Oxford University Press
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Rights and permissions
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

• Mar 12, 2022
• 11 min read

Public Policy 101: The Stages of the Policy Process

Public policies are everywhere in today’s world, but their ubiquity is also why their definition is often elusive and the analysis of public policies tends to be complex. The Public Policy 101 series offers the reader several tools of analysis that help make sense of the complexity of public policies. This 101 series comprises eight different articles, each focusing on a different aspect, which should provide the reader with a framework of analysis to better understand the complex world of public policy-making.

What is public policy?

The stages of the policy process

Rationalist and constructivist ontology in public policy

An overview of the theories of the policy process

The public policy actors

The policy subsystem

Beyond national public policy

New approaches in public policy studies

The first instalment of the series offered some definitions of public policy, highlighted its complexity, and concluded by suggesting that scholars use different heuristic tools to simplify the analysis of the public policy-making process. This article provides an overview of one of the earliest and most popular approaches: the disaggregation of the policy-making process in a series of discrete stages, known as the ‘policy cycle’.

The idea of a policy cycle stems from a common understanding of public policy as a ‘process’, as the definitions in the first instalment made clear. Policy outputs are shaped by the actions of policy-makers , operating within institutional structures on the basis of the ideas they hold regarding the problem and the potential solutions. Thus, disaggregating this process into multiple, interrelated stages through which policy flows in a more or less sequential fashion from inputs to outputs is a way to provide order and reduce analytical difficulties (see Howlett et al ., 2020: 8-9).

One of the first attempts to categorise the policy process came from Harold Lasswell, an early pioneer of the policy sciences. Lasswell (1956) divided the process into seven different stages, each with a specific policy-making function: intelligence, recommendation, prescription, invocation, application, appraisal, and termination. In the following decades, scholars built on this approach to stylise new and more empirically accurate models (e.g. Jones, 1970; Anderson, 1975; Brewer & deLeon, 1983), eventually settling with what is today called the ‘textbook approach’ (Nakamura, 1987).

This approach to the policy cycle broadly draws on the functions Lasswell (1956) assumed public policy-making ought to perform to identify five (sometimes six) stages of the policy process: (1) agenda setting, which recognises the problem; (2) policy formulation, where a solution is proposed; (3) decision-making, where the solution is chosen and legitimised; (4) implementation, in which the solution is put into action; (5) evaluation, or the monitoring of the results; and in some cases (6) the choice to either maintain, replace or terminate the policy. Below, each stage is delineated, and the advantages and disadvantages of the policy cycle model are discussed.

Agenda setting is the first in both logical and chronological order: the government can pass no policy if a problem is not identified in the first place. Agenda setting is concerned with the way policy problems emerge and how they gain the government’s attention (Howlett et al. , 2020: 100; see also Birkland, 2007: 63). In other words, the agenda-setting process aims to answer the question of ‘what makes an idea’s time come’ (Kingdon, 1984).

Agenda-setting scholars identify three ways through which items can reach the government’s agenda. Firstly, society can learn about problems through objective indicators. Examples are the rate of unemployment, inflation, pollution levels or criminality. These measures may indicate that things are getting worse, and that action is needed, thus making the issue gain considerable attention. Secondly, focusing events – sudden, relatively rare events that spark media and public attention (Birkland, 1997) – are another way to gain policy-makers’ attention. Examples are natural calamities, wars, or scandals. In the wake of such events, policy-makers may be pushed to provide an immediate solution, which requires giving heightened attention to the issue. Finally, since attention is scarce compared to the number of potential issues, agenda setting is fundamentally a competition to exercise power and define policy issues to establish their severity and causes (Cairney, 2019: 28). Agenda priorities thus create political winners and losers by their very nature (Zahariadis, 2016: 3). As such, for a problem to gain attention, it is important to create compelling stories that help policy-makers focus their attention to that particular issue (Peterson & Jones, 2016; Stone, 2012).

In sum, the agenda-setting process shows very well both the social construction of policy problems and the importance of power in policy-making. Zahariadis (2016) neatly summarises this stage using the so-called ‘Four Ps of agenda setting’: power, perception, potency (i.e. severity) and proximity.

Once the problem has been identified and put on the decision agenda, the next question to ask is: what is the plan for dealing with the problem (Sidney, 2007)? This is what the formulation stage sets out to do. This stage is essentially a matter of policy design: it is about setting objectives, choosing which course of action to take among the various options available, and which tools can be employed to address the problem (Howlett, 2018: 95ff.). The main job of policy formulation is then to ‘narrow down the range of all possible options to those that are available and that decision-makers might accept’ (Howlett et al. , 2020: 133). At this stage, policy-makers may avail themselves of the opinion of epistemic communities, stakeholders and interest groups, thus engaging in what is known as ‘evidence-based policy-making’ (see Cairney, 2016).

In the decision-making stage, legislators follow up on the formulated policies to legitimise them. Although ideally the decision to be made would be optimal and the result of an ‘evidence-based’ and informed choice, most often decision-making is a satisficing process, depending on institutional constraints and political calculations (Howlett & Giest, 2018). Hence, like agenda setting creates winners and losers in the selection of a problem, the decision-making stage creates winners and losers in the selection of the solution to be enacted (Howlett et al. , 2020: 177).

It is during the implementation stage that the intentions of the policy-makers are translated into action (Barrett, 2004: 255). Implementation is arguably the most crucial stage since a policy on paper is not worth anything unless it is implemented. But when can a policy be said to be ‘properly implemented’? For a long time, scholars took a ‘top-down’ approach whereby policies are said to either fail or succeed based on several factors, such as the clarity of the policy’s objectives, the presence of financial and ideational resources and of skilled and compliant officers, minimal dependency relationship (i.e. few ‘veto players’), and generally few external constraints beyond the control of policy-makers (Cairney, 2019: 28-9; see also Pressman & Wildavsky, 1973; Sabatier & Mazmanian, 1980).

This ‘top-down’ school was opposed to a ‘bottom-up’ approach whereby central decision-makers only have a limited role, since implementation mainly relies on a decentralised apparatus of civil servants and administrative officials (Howlett et al. , 2020: 210). Bottom-uppers identify the main actors of policy delivery in local bureaucrats and see implementation as a ‘decentralised problem-solving’ exercise where negotiations take place within networks of implementers (Pülzl and Treib, 2007; see also Hjern & Hull, 1982; Thomann, 2019: 4). Several attempts have been made to synthesise the two positions (e.g. Matland, 1995; Sabatier, 1986; Winter, 2012), but the two perspectives essentially reflect preferences in research design: top-downers take a prescriptive approach, whereas bottom-uppers a descriptive one (Cairney, 2019: 29). Neither, alone, offers a comprehensive view of implementation.

Policy evaluation is about assessing the extent to which a policy was successful (Cairney, 2019: 32; see also Howlett et al., 2020: 241; Jann & Wegrich, 2007: 54). It is the last and most contentious stage of the policy process for two reasons. First, it is very difficult to answer the question ‘to what degree was the policy successful?’ since it requires knowing whose success is being measured (the government’s? the stakeholders’? the lobbyists’?), and about what kind of success policy-makers should care (the effect on their popularity? the completion of their agenda? the long-term impacts?) (Cairney, 2019: 32; see also McConnell, 2010). Hence, evaluation is by necessity subjective and very difficult to carry out analytically. Secondly, what is significant at this stage is not so much the success or failure of the policy, but the ‘lesson-drawing’ aspect for policy-makers. Evaluation should be an opportunity for policy-makers to shed light on the experience of the policies on the ground to understand whether or not the intended results were achieved and what can be learned for future policies (Howlett et al. , 2020: 242; Howlett & Giest, 2018: 24).

At the end of this process, and some time after the policy is enacted, legislators must decide whether to maintain , replace or terminate it. A policy, for instance, may be terminated either because it has absolved its goals, or because it has been proven to be ineffective (Jann & Wegrich, 2007: 55). But under real-world circumstances, this is a rather difficult process, because of the path-dependent nature of policies. Governments tend to inherit policies from past administrations, and they generally accept the ongoing programmes they find when coming into office, since it would be too costly (both economically and politically) to reverse them (John, 2013: 25; on inheritance, see Rose, 1990; on path dependence, see Pierson, 2004). Hence, termination is often not included in discussions of the policy cycle both because of its relative rarity and because, unlike the other stages, it is not always necessary.

It is clear from the discussion so far that the policy cycle model has some important advantages. First of all, and most apparently, this disaggregation facilitates the understanding of multi-dimensional processes by bringing logical and chronological order into what is a very complex web of interactions. Secondly, the policy cycle model can be applied at multiple levels of analysis, ranging from the local to the international. For instance, Mattli and Woods (2009) apply a similar model, called ANIME – Agenda setting, Negotiations, Implementation, Monitoring, and Enforcement – to describe patterns of international public policy among states. Finally, the policy cycle model addresses the intertwining of actors, institutions and ideas by describing where action takes place, who the actors are, and what they strive for.

However, the disadvantages perhaps outweigh the advantages of this model, which explains why most public policy textbooks today eschew such an approach (see for instance Cairney, 2019: 33-5; Howlett et al. , 2020: 12-3; John, 2013:18-20 for critiques). Firstly, its sequentiality suggests an apparent linearity that does not, in fact, exist in most real-world cases (John, 2013: 18-20). For instance, it is widely acknowledged that the implementation stage can be used to rethink and reformulate policies (John, 2013: 20; Thomann, 2019). Secondly, the policy cycle model lacks any notion of causation: it does not explain what, or who, drives a policy stage from one stage to another nor why this should be the case. Finally, this model remains silent about the content of a policy and how this may affect policy-making styles (see Howlett et al. , 2020: 13).

Hence, while the policy cycle model is often one of the starting points in public policy courses, two correctives should be applied. First, its apparent linearity should be taken writ large in the minimal sense that a formal policy must be proposed and legislated on, and the means of its implementation agreed upon. Such a model is more relevant for understanding the presentation and legitimation of policy rather than detecting the bargaining that happens during decision-making and implementation (John, 2013: 19-20). Secondly, rather than a model, the policy cycle should be seen as a methodologic heuristic founded upon a deep empirical record, which can serve as a benchmark for two different tasks. It can offer a perspective against which the democratic quality of the processes of policy can be assessed (Jann & Wegrich, 2007: 58). And, by facilitating the investigation of the public policy process, this approach can help with the accumulation of empirical insights from multiple cases that can be aggregated based on the stage at which they are analysed in order to generate new models and theories of the policy process (Howlett et al. , 2020: 274). The next steps of this series will be to understand the ontology on which public policy theories can rest.

Anderson, J. E. (1975). Public Policymaking . New York, NY: Praeger.

Barrett, S. M. (2004). Implementation studies: time for a revival? Personal reflections on 20 years of implementation studies. Public Administration , 82 (2), 249-262.

Birkland, T. A. (1997). After disaster: Agenda setting, public policy, and focusing events . Washington, DC: Georgetown University Press.

Birkland, T. A. (2007). Agenda Setting in Public Policy. In Handbook of Public Policy Analysis – Theory, Politics, and Methods . Fischer, F., Miller G. J., and Sidney, M. S. (Eds.). London: CRC Press, pp. 63-78.

Brewer, G., and deLeon, P. (1983). The Foundations of Policy Analysis . Monterey, CA: Brooks, Cole.

Cairney, P. (2016). The Politics of Evidence-Based Policymaking. London: Palgrave Pivot.

Cairney, P. (2019). Understanding Public Policy. 2nd edition. London: Red Globe Press.

Hjern, B., & Hull, C. (1982). Implementation research as empirical constitutionalism. European Journal of Political Research , 10 (2), 105-115.

Howlett, M. (2018). Designing Public Policies. Principles and Instruments. 2nd edition. London & New York, NY: Routledge.

Howlett, M. and Giest, S. (2018). The policy-making process. In Routledge Handbook of Public Policy. Araral, E. Jr, Fritzen, S., Howlett, M. and Ramesh, M. (Eds.). London and New York, NY: Routledge, pp. 17-28.

Howlett, M., Ramesh M., and Perl, A. (2020). Studying Public Policy. Principles and Processes . 4th edition. Oxford: Oxford University Press.

Jann, W., and Wegrich, K. (2007). Theories of the Policy Cycle. In Handbook of Public Policy Analysis – Theory, Politics, and Methods . Fischer, F., Miller G. J., and Sidney, M. S. (Eds.). London: CRC Press, pp. 43-62.

John, P. (2013). Analyzing Public Policy . 2nd edition. London & New York, NY: Routledge.

Jones, C. (1970) An Introduction to the Study of Political Life . 3rd edition. Berkeley, CA: Duxberry Press.

Kingdon, J. (1984). Agendas, Alternatives and Public Policies. New York, NY: Harper Collins.

Lasswell, H.D. (1956). The Decision Process: Seven Categories of Functional Analysis . College Park, MD: University of Maryland Press.

Matland, R. E. (1995). Synthesizing the implementation literature: The ambiguity-conflict model of policy implementation. Journal of Public Administration Research and Rheory , 5 (2), 145-174.

Mattli, W. and Woods, N. (2009). The Politics of Global Regulation . Princeton, NJ: Princeton University Press.

McConnell, A. (2010). Understanding Policy Success. Rethinking Public Policy. New York, NY: Palgrave MacMillan.

Nakamura, R.T. (1987). The Textbook Policy Process and Implementation Research. Policy Studies Review , 7 , 142–154.

Peterson, H. L., & Jones, M. D. (2016). Making sense of complexity: the narrative policy framework and agenda setting. In Handbook of Public Policy Agenda Setting . Zahariadis, N. (Ed.). Cheltenham, UK and Northampton, MA: Edward Elgar Publishing, pp. 106-131.

Pierson, P. (2004). Politics in Time: History, Institutions and Social Analysis. Princeton, NJ: Princeton University Press.

Pressman, J. and Wildavsky, A. (1973). Implementation. How great expectations in Washington are dashed in Oakland; or why it’s amazing that federal programs work at all. Berkeley, CA: University of California Press.

Pülzl, H. and Treib, O. (2007). Implementing Public Policy. In Handbook of Public Policy Analysis – Theory, Politics, and Methods . Fischer, F., Miller G. J., and Sidney, M. S. (Eds.). London: CRC Press, pp. 89-108.

Rose, R. (1990). Inheritance before choice in public policy. Journal of Theoretical Politics , 2 (3), 263-291.

Sabatier, P. A. (1986). Top-down and bottom-up approaches to implementation research: a critical analysis and suggested synthesis. Journal of Public Policy , 6 (1), 21-48.

Sabatier, P. A. and Mazmanian, D. (1980). The implementation of public policy: A framework of analysis. Policy Studies Journal 8 (4):538-560.

Sidney, M. S. (2007). Policy Formulation: Design and Tools. In Handbook of Public Policy Analysis – Theory, Politics, and Methods . Fischer, F., Miller G. J., and Sidney, M. S. (Eds.). London: CRC Press, pp. 79-87.

Stone, D. (2012). Policy Paradox: The Art of Political Decision Making . New York, NY: W. W. Norton & Company.

Thomann, E. (2019). Customized implementation of European Union food safety policy: United in diversity? Cham, Switzerland: Palgrave Macmillan.

Winter, S. C. (2012). Implementation Perspectives: Status and Reconsideration. In The SAGE Handbook of Public Administration , 2nd edition, Peters, B. G. and Pierre, J. (Eds.). London: SAGE Publications, pp. 265-278.

Zahariadis, N. (2016). Setting the agenda on agenda setting: definitions, concepts and controversies. In Handbook of Public Policy Agenda Setting . Zahariadis, N. (Ed.). Chelthenam and Northampton, MA: Edward Elgar Publishing, pp. 1-22.

Image References

Figure 1: BBC. (2017, December 4th). Picture from the manuscript Zoroaster Clavis Artis, MS. Verginelli-Rota, Biblioteca dell'Accademia Nazionale dei Lincei, Rome (Italy) [Drawing]. Retrieved from: https://www.bbc.com/culture/article/20171204-the-ancient-symbol-that-spanned-millennia

Figure 2: Own elaboration.

Figure 3: Shutterstock. (n.d.). Multiracial fists hands up vector illustration. Concept of unity, protest, revolution, fight, cooperation. [Flat outline design]. Retrieved from: https://www.shutterstock.com/image-vector/vector-illustration-different-hands-white-bannerss-1287073093?irclickid=2gMwztR7WxyIUoL3yWzwET6DUkGWY4zGi2fkzo0&irgwc=1&utm_medium=Affiliate&utm_campaign=TinEye&utm_source=77643&utm_term=&c3ch=Affiliate&c3nid=IR-77643

Figure 4: Shutterstock. (n.d.). Overhead Railroad Crossing [Photograph]. Retrieved from: https://www.shutterstock.com/image-photo/overhead-railroad-crossing-86284846?irclickid=2gMwztR7WxyIUoL3yWzwET6DUkGWY9QHi2fkzo0&irgwc=1&utm_medium=Affiliate&utm_campaign=TinEye&utm_source=77643&utm_term=&c3ch=Affiliate&c3nid=IR-77643

Figure 5: Own elaboration.

Marco Schito

Arcadia has an extensive catalog of articles on everything from literature to science — all available for free! If you liked this article and would like to read more, subscribe below and click the “Read More” button to discover a world of unique content.

Let the posts come to you!

Thanks for submitting!

Policy The policy cycle

European Geosciences Union

The policy cycle

One way to improve the impact of your scientific research is to engage with policy. Doing so can create new opportunities for yourself and your research. The main challenges are knowing when and how to effectively communicate scientific results to policy. If the wrong timing or communication method is chosen then results are less likely to be incorporated into the policy process.

The policy cycle is an idealised process that explains how policy should be drafted, implemented and assessed. It serves more as an instructive guide for those new to policy than as a practical strictly-defined process, but many organisations aim to complete policies using the policy cycle as an optimal model.

The policy cycle is made up of roughly six stages and science can be incorporated into every step. How science supports these different stages are described below.

Agenda setting

This step identifies new issues that may require government action. If multiple areas are identified they can all be assessed, or particular issues may be given a priority.

Scientific Input : New scientific results can be the foundation for forming new policies. Additionally, new focus areas can be anticipated through horizon / foresight scanning events that aim to identify emerging issues of policy-relevance.

Example : a foresight study may indicate that the growing population and steadily increasing energy consumption per capita will require an increased energy production. This, along with the need to reduce emissions and limit future climate change, may result in policymakers deciding to increase solar panel production and usage.

Formulation This step defines the structure of the policy. What goals need to be achieved? Will there be additional implications? What will the costs be? How will key stakeholders react to these effects? Scientific Input : Science can be incorporated in this stage through impact assessments, which aim to comprehensively assess what effects will occur from a potential policy. These assessments can study multiple strategies to identify the optimum policy. Example : Should governments offer tax-breaks to start-up renewable energy companies? Or should they offer individual subsidies to solar panel buyers? What might be the effects of these actions? Adoption

Once the appropriate approval (governmental, legislative, referendum voting etc.) is granted then a policy can be adopted.

Science Input : Those in charge of approving a certain policy will often seek external advice that is independent to those who drafted the policy. Scientists can be called upon to offer advice within the decision-making process.

Example : A nation-wide policy to increase solar capacity can be implemented by the national government, but changing a law will require a vote in Parliament.

Implementation

Establishing that the correct partners have the resources and knowledge to implement the policy. This could involve creating an external organisation to carry out actions. Monitoring to ensure correct policy implementation is also necessary.

Scientific Input : Scientific advice can logistically support the policy being implemented. Scientists can provide methodological guidance to policy workers and advisory bodies who implement the policy.

Example : Administration processes to allow organisations and individuals to apply for solar energy subsidies / tax benefits need to be created.

This step assesses the effectiveness and success of the policy. Did any unpredicted effects occur? These assessments can be quantitative and/or qualitative.

Scientific Input : Scientists can evaluate the efficiency and effectiveness of policies. This can be done independently or working with policy implementers.

Example : The UK and Germany introduced highly popular solar energy policies. Energy production at certain times of the day and year have substantially increased. Occasionally more energy is being produced than is needed, which leads to further questions about how to handle the 'excess' energy.

Support / maintenance

This step studies how the policy might be developed, or provides additional support for its continuation. Additionally, the policy can be terminated if deemed redundant, accomplished, or ineffective.

Scientific Input : As a policy is continued, scientific advice may be needed on an ad-hoc basis. Updated feedback can be given when needed to help maintain and improve policies.

Example : Even if a policy is considered a success, should it be continued? Should solar panel policies be continued, or should policies now focus on improving national electric grids, or should energy storage policies be developed instead?

Remember that scientists should only offer a supportive role to the policy cycle. They should present only the current state of scientific knowledge. Policy officials are the decision-makers.

The policy cycle has been described as a theoretical concept that it not fully translatable to real world applications. Sometimes, some stages of the cycle are never delivered. Without scientists some of the stages are difficult to accomplish, therefore scientists are in a position to strengthen the policy cycle’s structure through expert advice and assistance.

• Policy Concepts in 1000 Words: The Policy Cycle and its Stages
• ‘Working at the science policy interface’ presentation by Panos Panagos

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

•  We're Hiring!
•  Help Center

Policy Cycle

• Most Cited Papers
• Most Downloaded Papers
• Newest Papers
• Relations Between School and Family Follow Following
• Production theory Follow Following
• Asiatic mode of production Follow Following
• Policy formulation Follow Following
• Karl Marx Theory of the Mode of Production Follow Following
• Population Policy Follow Following
• Educational Policy Studies Follow Following
• Biopolitics Follow Following
• Macedonia Follow Following
• Global Leadership Follow Following

Enter the email address you signed up with and we'll email you a reset link.

• Academia.edu Journals
•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

Writing a Research Paper

The information search process, some definitions.

• Choosing a Topic and Identifying Keywords
• Finding Library Materials: Using LibGuides
• Using HillSearch (Library Catalog) to Find Books, Ebooks, And Streaming Media
• Search Databases for Scholarly Journal Articles
• Synthesizing Sources
• Citing Your Research/Creating a Bibliography
• Where to Get Help with Research and Writing

Your Librarian

Have you been assigned a research paper or presentation and are uncertain where to begin? Librarians are here to help you at each step of the research process, from initial topic selection to preparing your bibliography. In her work, Rutgers Information Science Professor Carol Kuhthau defines the steps of  the  Information Search Process  and the feelings researchers experience during each part of it -including occasional feelings of frustration and discouragement. Recognizing that frustration can be part of the research process can help you in the long run. Grappling with initial discomfort can ultimately help you develop your thesis statement, craft arguments and find the resources that best meet your research needs..

The chart below describes The Information Search Process, the tasks related to each step of the process, and the related feelings you may experience.

We're happy to meet with you one-on-one to help you with your research and compiling your citations. Click on the "Book an Appointment" link in our profile boxes to set up a meeting. If you don't see a time that meets your needs, email us.

Your professor might require a specific type or number of sources for your assignment. The following definitions might be of help understanding the assignment. These definitions have been adapted from the Online Dictionary of Library and Information Science .

Abstract -  A brief, objective representation of the essential content of a book, article, speech, report, dissertation, patent, standard, or other work, presenting the main points in the same order as the original but having no independent literary value. A well-prepared abstract enables the reader to 1) quickly identify the basic content of the document, 2) determine its relevance to their interests, and 3) decide whether it is worth their time to read the entire document

Annotated Bibliography -  A bibliography in which a brief explanatory or evaluative note is added to each reference/citation and abstract. An annotation can be helpful to the researcher in evaluating whether the source is relevant to a given topic or line of inquiry. For more information, watch our video tutorial on creating an annotated bibliography .

Primary Source (non/science topics)  -  A  document or record containing firsthand information or original data on a topic, used in preparing a new (derivative) work. Primary sources include original manuscripts, periodical articles reporting original research or thought, diaries, memoirs, letters, journals, photographs, drawings, posters, film footage, sheet music, songs, interviews, government documents, public records, eyewitness accounts, newspaper clippings, etc.

Primary Source/Primary Study (science topics) -  Also called empirical research studies, primary research studies in the sciences report on an experiment that was performed by the author(s) of the study.  These articles are formatted similarly to a lab report, and will contain the following sections: Abstract, Introduction, Methods, Results, Discussion, and References.  Primary research studies will often contain data tables, graphs, and statistical analyses.

Scholarly Journal - A journal  publishing original research and commentary on current developments in a specific discipline, subdiscipline, or field of study (example: Journal of Clinical Epidemiology ). Scholarly journals are usually published in quarterly, bimonthly, or monthly issues sold by subscription (click here to see an example). Articles in scholarly journals are usually written by the person (or persons) who conducted the research. Longer than most magazine articles, they almost always include a bibliography or list of works cited at the end. In journals in the sciences and social sciences, an abstract usually precedes the text of the article, summarizing its content. Most scholarly journals are peer-reviewed, meaning article drafts are reviewed by a panel of experts prior to publication and any needed edits are made by the author. Not all periodicals are scholarly. Some are popular magazines - such as Time or People . Other periodicals are produced for a particular discipline - such as Inc. or Education Week - but articles are written by journalists, not disciplinary experts.

• Next: Choosing a Topic and Identifying Keywords >>
• Last Updated: Jul 15, 2024 10:53 AM
• URL: https://libguides.stonehill.edu/writing_a_research_paper

Global academic research is skewed toward rich countries. How do World Bank policy research papers fit in?

Brian stacy, quy-toan do, deon filmer.

Academic research is a major source of insights from data. These insights can be critical in helping policymakers manage public resources, formulate policies, and help the public understand the world around them. However, academic research is not evenly distributed across countries. As previous research has shown, wealthier countries are more likely to be studied by academics.  The World Bank is a major source of academic research. Policy Research Working Papers (PRWPs) are a key output of the World Bank. These papers aim to provide insights to policymakers in World Bank client countries, which are mostly low- and middle-income countries. How do these World Bank publications fare in terms of filling gaps in empirical academic research for World Bank clients? To examine this, we build on the approach of Stacy, Kitzmüller, Wang, Mahler, & Serajuddin (2024) to classify empirical academic articles based on data use. We compare the number of empirical academic articles and World Bank policy research working papers by country. What did we find? Scroll below to learn more.  

Do PRWPs fill gaps in empirical research for World Bank clients? To some extent yes! By focusing more on low- and middle-income countries, PRWPs help fill critical gaps left by other empirical studies. However, while they contribute to a more equitable distribution of research, the challenge of fully correcting the skew towards wealthier countries remains. A lack of high quality, timely, and open data sources is major issue in several low- and middle-income countries. The continued production and dissemination of PRWPs are vital steps toward a more inclusive and comprehensive understanding of global development issues.

Get updates from Data Blog

Thank you for choosing to be part of the Data Blog community!

Your subscription is now active. The latest blog posts and blog-related announcements will be delivered directly to your email inbox. You may unsubscribe at any time.

Data Scientist, Development Data Group, World Bank

Co-Director, World Development Report 2023

Director, Development Research Group, World Bank

Join the Conversation

• Share on mail
• comments added

Automation, Career Values, and Political Preferences

Career opportunities and expectations shape people’s decisions and can diminish over time. In this paper, we study the career implications of automation and robotization using a novel data set of resumes from approximately 16 million individuals from the United States. We calculate the lifetime "career value" of various occupations, combining (1) the likelihood of future transitions to other occupations, and (2) the earning potential of these occupations. We first document a downward trend in the growth of career values in the U.S. between 2000 and 2016. While wage growth slows down over this time period, the decline in the average career value growth is mainly due to reduced upward occupational mobility. We find that robotization contributes to the decline of average local labor market career values. One additional robot per 1000 workers decreased the average local market career value by $3.9K between 2004 and 2008 and by $2.48K between 2008 and 2016, corresponding to 1.7% and 1.1% of the average career values from the year 2000. In commuting zones that have been more exposed to robots, the average career value has declined further between 2000 and 2016. This decline was more pronounced for low-skilled individuals, with a substantial part of the decline coming from their reduced upward mobility. We document that other sources of mobility mitigate the negative effects of automation on career values. We also show that the changes in career values are predictive of investment in long-term outcomes, such as investment into schooling and housing, and voting for a populist candidate, as proxied by the vote share of Trump in 2016. We also find further evidence that automation affected both the demand side and supply side of politics.

Maria Petrova thanks funding from the European Research Council (ERC) under the European Union Horizon 2020 research and innovation program (Grant Agreement 803506), the Plan Nacional project PID2020120118GBI0, and the Severo Ochoa Programme for Centres of Excellence in R&D (Barcelona School of Economics CEX2019-000915-S), funded by MCIN/AEI/10.13039/501100011033. Yildirim thanks the Mack Institute and the Dean’s Research fund at the Wharton School for their research support. The views expressed herein are those of the authors and do not necessarily reflect the views of the National Bureau of Economic Research.

MARC RIS BibTeΧ

Download Citation Data

More from NBER

In addition to working papers , the NBER disseminates affiliates’ latest findings through a range of free periodicals — the NBER Reporter , the NBER Digest , the Bulletin on Retirement and Disability , the Bulletin on Health , and the Bulletin on Entrepreneurship  — as well as online conference reports , video lectures , and interviews .

• Open access
• Published: 15 July 2024

Theory of change for addressing sex and gender bias, invisibility and exclusion in Australian health and medical research, policy and practice

• Thomas Gadsden 1   na1 ,
• Laura Hallam 1 , 2   na1 ,
• Cheryl Carcel 1 ,
• Robyn Norton 1 , 3 ,
• Mark Woodward 1 , 3 ,
• Louise Chappell 4 &
• Laura E. Downey   ORCID: orcid.org/0000-0002-9563-7113 1 , 3  

Health Research Policy and Systems volume  22 , Article number:  86 ( 2024 ) Cite this article

Metrics details

Sex and gender are inadequately considered in health and medical research, policy and practice, leading to preventable disparities in health and wellbeing. Several global institutions, journals, and funding bodies have developed policies and guidelines to improve the inclusion of diverse participants and consideration of sex and gender in research design and reporting and the delivery of clinical care. However, according to recent evaluations, these policies have had limited impact on the inclusion of diverse research participants, adequate reporting of sex and gender data and reducing preventable inequities in access to, and quality provision of, healthcare. In Australia, the Sex and Gender Policies in Medical Research (SGPMR) project aims to address sex and gender bias in health and medical research by (i) examining how sex and gender are currently considered in Australian research policy and practice; (ii) working with stakeholders to develop policy interventions; and (iii) understanding the wider impacts, including economic, of improved sex and gender consideration in Australian health and medical research. In this paper we describe the development of a theory of change (ToC) for the SGPMR project. The ToC evolved from a two-stage process consisting of key stakeholder interviews and a consultation event. The ToC aims to identify the pathways to impact from improved consideration of sex and gender in health and medical research, policy and practice, and highlight how key activities and policy levers can lead to improvements in clinical practice and health outcomes. In describing the development of the ToC, we present an entirely novel framework for outlining how sex and gender can be appropriately considered within the confines of health and medical research, policy and practice.

Peer Review reports

Contributions to the literature

Inadequate consideration of sex and gender in health and medical research, policy and practice contributes to inequity in the distribution of health.

No framework for guiding the consideration of sex and gender in health and medical research, policy and practice currently exists.

The theory of change presented in this paper presents an entirely novel and important theoretical scaffold for institutions and organizations to use when considering how to actively enhance sex and gender awareness and inclusivity in their activity.

Reorienting practice through informed action can contribute to improving population health and economic wellbeing by enhancing sex and gender equity.

Introduction

Sex and gender are important determinants of health but are often inadequately considered in health and medical research, policy and practice [ 1 , 2 , 3 ]. Documented issues include exclusion or underrepresentation of cis women, trans women and men, those with intersex characteristics and non-binary populations in research participation and lack of sex and gender disaggregation and interpretation of data (see Table  1 for definitions of key concepts) [ 2 , 4 ]. As health and medical research forms the evidence base that many clinical guidelines and health policies are based on, this can lead to disparities in diagnosis, treatment and health outcomes. Sex- and gender-based health disparities have been identified in many conditions, such as cardiovascular disease [ 5 , 6 ], stroke [ 7 ] and pain [ 8 ], and heath areas such as screening, diagnosis [ 9 ] and reactions to pharmacological treatments [ 10 ].

In response to these health disparities, in recent decades several research institutions have developed policies and guidelines to improve the inclusion of diverse participants and consideration of sex and gender in research design and reporting [ 11 , 12 , 13 , 14 , 15 ]. New and updated policies increasingly highlight broader issues of equity and intersectionality alongside sex and gender considerations [ 16 ]. The aims and potential impacts of these policies include improved rigour, ethics and reproducibility of science, counteracting existing biases and exclusionary practices, understanding health differences and inequities, informing and improving health policy and care and advancing gender equality, diversity and inclusion. Positive initiatives towards updating university science and medical curricula [ 17 , 18 , 19 ], clinical protocols and guidelines [ 6 , 20 ] and public and global health programs to explicitly incorporate sex and gender considerations have also been reported [ 21 , 22 ].

Evaluations of these policies and guidelines report mixed results on health and medical research design and reporting to date [ 24 , 25 , 26 ]. For instance, an evaluation of the Canadian Sex- and Gender-Based Analysis policy found increased explicit consideration of sex and gender in grant applications [ 27 ], while other policy evaluations have found limited impact on the inclusion of women and marginalized community members in study populations and on the disaggregation and analysis of results by sex and gender [ 28 , 29 , 30 ]. A range of barriers to progress have also been reported, including limited awareness, a lack of training and resources among personnel across the medical research pipeline, and the absence of adequate accountability and monitoring mechanisms by regulators, including government agencies, funders and universities, among others [ 31 , 32 , 33 ].

In Australia, the Sex and Gender Policies in Medical Research (SGPMR) project was established to understand the current state of play in relation to the explicit consideration of sex and gender in medical research and practice [ 34 ]. SGPMR is a philanthropically funded initiative with three primary aims: (i) to understand whether and how sex and gender are addressed in current research policy and practice in Australia; (ii) to work with stakeholders to co-develop policy interventions; and (iii) to understand the wider impacts, including economic impacts, of improving sex and gender consideration in Australian health and medical research [ 35 ]. Work to date has demonstrated that sex and gender are under-reported in research articles published in Australia’s top ten medical journals in 2020 [ 3 ] and that the content of academic journals dedicated to women’s health remains largely focussed on reproductive health topics, with few articles targeting the major causes of morbidity and mortality in women [ 36 ].

Common to each of these disparate initiatives is an implied shared understanding in the notion that explicit consideration of sex and gender in research design, policy and operations leads to better data and evidence-based practice, which in turn leads to better health outcomes. As has been previously argued [ 37 ], a clearly articulated explicit framework outlining the causal pathways by which better data and gender-sensitive practice would lead to better health would ensure that the opportunity for misunderstanding is minimized and enhance the coordination of efforts to achieve common goals and the ability to create and evaluate their impact enhanced. A Theory of Change (ToC), defined as “an explicit process of thinking through and documenting how a program or intervention is supposed to work, why it will work, who it will benefit and the conditions required for success”, is an increasingly common means in which to articulate a shared vision and map logical pathways to impact towards addressing a problem or issue [ 38 ]. This methodology is increasingly used in public health and evaluation frameworks to articulate how an intervention can achieve long-term impact by identifying and depicting causal pathways from activities to outputs and outcomes and the key mechanisms, barriers, and facilitators underpinning these causal pathways [ 39 , 40 , 41 , 42 ]. Although ToCs are typically developed for discrete projects or interventions, they have been used to promote shared ownership and understanding among stakeholders for broader initiatives, such as strengthening sector-wide response to human immunodeficiency virus (HIV) in Papua New Guinea [ 43 ] and multi-sector urban planning initiatives [ 44 ].

In this paper, we outline the development of a ToC to identify the pathways through which improved consideration of sex and gender in health and medical research, policy and practice could impact social and economic health outcomes. The objectives of this work are twofold: (1) to fill an important knowledge-implementation gap in the literature by explicitly documenting the problem, the desired impact and how engaging in certain activities can contribute towards achievement of positive change across the evidence, policy and practice pipeline; and (2) to situate the activities of the aforementioned SGPMR project in this wider context to guide future project activities aimed at creating impact in the Australian health and medical research sector.

Study design

This study followed best practice guidelines for developing a ToC, involving a wide range of stakeholders and end-users, ensuring rigorous evidence-based discussions through participatory research methods and engaging in an iterative process of refinement [ 45 , 46 ]. These recommendations are reflected by the iterative six-step process followed in this study: (1) initial mapping of key concepts and considerations; (2) stakeholder interviews; (3) draft ToC development; (4) stakeholder consultation workshop; (5) revised ToC development; and (6) stakeholder review. Table 2 outlines the goals and methods of each of the steps. For transparency, this study is reported against the Standards for Reporting Qualitative Research (Supplementary file 1) and the Checklist for reporting ToC in Public Health Interventions (Supplementary file 2) [ 45 ].

Data collection

Initial mapping.

A horizon scanning exercise was undertaken to identify literature of importance to this project by using PubMed and Google Scholar searches with different combinations of the key words “sex”, “gender”, “health” “medical” “research” “policy” “framework” “logic” and “theory of change”. Titles and abstracts were screened and full papers reviewed for literature identified as potentially relevant. No evidence was identified that directly addressed the development of a framework for explicit consideration of sex and gender in medical research, policy and practice. Literature on the development and use of a ToC was reviewed to determine the most appropriate structural framework for the purposes of this ToC. The approach chosen draws upon programmatic theory described as “deal[ing] with the mechanisms that intervene between the delivery of a program service and the occurrence of outcomes of interest” [ 29 ]. This approach requires that all activities and their intended outputs, outcomes and impacts are identified and then mapped to a ToC structural framework. Table 3 defines each component of the ToC structural framework used in this exercise. Barriers and facilitators to reaching the intended outcomes and impacts were also considered.

Study participants

Participation in the study was restricted to members of the SGPMR project [ 34 ]. There are 24 SGPMR project members in total: 8 principal investigators (PI) and an advisory group of 16 members. Members represent government, cisgender, trans, intersex, non-binary and indigenous community groups, as well as multidisciplinary academics with expertise in health and medicine, gender, human rights, policy, clinical care, regulation and community engagement.

Semi-structured interviews

All 24 individuals associated with the SGPMR project were invited via email to participate in a semi-structured interview. Interviews were conducted by one of three members of the core ToC research team (L.D., L.H., T.G.) who all have experience in qualitative research. Semi-structured interviews of 40–60 min were conducted, either online, via Zoom or in person. The predetermined interview guide included a brief introduction to ToC methodology and questions were organized around the structural framework to obtain views on the key problems, activities, outputs, outcomes and impacts that related to the SGPMR project and sex and gender in the health and medical research sector (Supplementary file 3). Questions were primarily framed in relation to the SGPMR project, thereby concerning the Australian context.

Interviews were audio recorded, with the written consent of participants. The research team reviewed the audio transcripts produced by the Zoom transcribe function alongside the audio recordings to develop an accurate transcript for each interview. Interview transcripts were coded using NVivo 12. Interview responses were mapped deductively to the elements of the structural framework and analysed inductively to identify common themes across interviews. Codes were discussed between the core ToC research team iteratively and the final list of codes was used to develop the draft ToC. Codes addressing similar themes were combined where possible.

Consultation workshop

All 24 members of the SGPMR project, regardless of whether they participated in an interview, were invited to provide further input on the draft ToC by participating in a 2-h online workshop on 23 September 2022. The draft ToC was presented by the lead facilitator (L.D.) and each element was presented for discussion and feedback amongst the group. Content, language and structure of the draft were all reviewed, with further explanation by the facilitators and input from stakeholders. The discussion was audio-recorded with the consent of participants and one facilitator took extensive notes (T.G.), which were used by the research team to make amendments to the draft.

This study received ethical approval from the Human Research Ethical Approval Panel (HREAP) at the University of New South Wales (HC220443). All participants provided written consent to participate in this study.

Participation

A total of 15 individuals (4 PI members, 11 advisory group members), participated in semi-structured interviews, and 7 individuals (4 PI members, 3 advisory group members) participated in the 2-h online workshop. Participants represented expertise in clinical research, social sciences, academic, non-government community-based organizations and government. The most common reason for declining participation was unavailability. A total of 12 individuals (5 PI members and 7 advisory group members) provided peer review of the draft ToC schematic. In total, 19 out of 24 (79.2%) individuals associated with the SGPMR project provided some input into the ToC.

• Theory of change

The ToC is shown in Fig.  1 (see Supplementary File 4 for a tabulated version). Results are structured according to the elements of the ToC (problem statement, required activities, outputs and outcomes, impact). Boxes in the ToC are referred to in the results below numbered from left to right across the diagram for each element of the framework.

Theory of change for addressing sex and gender bias, invisibility and exclusion in health and medical research, policy and practice

Five key problems regarding the current consideration of sex and gender in health and medical research, policy and practice were identified for inclusion in the ToC: (1) lack of awareness of existing sex and gender biases and how those intersect with other biases in health and medical research, policy and practice; (2) Inadequate and biased incorporation of sex and gender into health and medical research; (3) sex and gender-based exclusion from meaningful engagement and participation in health and medical research; (4) lack of evidence-based interventions to address sex and gender biases in research, policy and practice; and (5) inequitable health outcomes between different populations.

Interviewees identified various forms of bias in research against different groups on the basis of sex and gender, which lead to science that is not rigorous or representative. Several potential reasons were proposed for inadequate consideration of sex and gender in research including poor societal understanding of sex and gender, particularly the predominance of a binary concept of sex and gender, inadequate data collection, lack of inclusion and analysis by sex and gender, too strong a focus on biological sex and a lack of consideration of intersectional factors, including race, social, economic and other factors. Other issues included the exclusion of marginalized communities from participating in research, a lack of adequate community consultation and input into research projects, which can lead to the design and funding of unethical research. Stakeholders discussed inequitable access to healthcare, which contributes to poorer health outcomes, particularly for women, transgender and gender-diverse people and people with intersex variations, and associated economic losses.

A total of 12 activities with potential to address the problems raised above emerged from participants’ responses. As a key starting point, respondents emphasized the need for improved understanding of the terms sex and gender, both within the health and medical research sector and societally. Respondents stressed that the complexity of these concepts and their evolving nature required a suitable conceptual framework that could be used to guide other activities. Accounting for intersectionality was also highlighted in the workshop as a vital component for any such framework to consider.

These themes are reflected in the ToC through the inclusion of activities that relate to building knowledge and awareness, education, training and advocacy:

Sector-wide discussion on conceptions of sex and gender and their relationship to intersectional factors;

Development and delivery of education and training on sex and gender concepts, their relevance and application to health research and translation;

Advocacy and awareness building around current issues and solutions;

Mapping current knowledge, policy and practice in relation to sex and gender.

Two activities focussed on the need for meaningful consultation and the building of networks and partnerships to share knowledge and expertise and facilitate change that appropriately accounts for the needs of diverse communities:

Diverse community and stakeholder involvement and engagement in evidence generation, translation, implementation and evaluation;

Developing diverse and multidisciplinary networks and communities of practice.

Several activities focussed on changing research practice and the development of policies, guidelines and standards to assist this change:

Policy development and implementation throughout the health and medical research sector;

Production or implementation of standards for consideration of sex and gender in research design;

Collection of accurate and inclusive sex- and gender-related and disaggregated data. Many stakeholders suggested that the Australian Bureau of Statistics Standard for Sex, Gender, Variations of Sex Characteristics and Sexual Orientation Variables [ 47 ] is a locally relevant example of such a standard that can be further implemented and used to guide data collection.

Expanding beyond health and medical research, two activities focussed on supporting translation of research evidence into practice:

Production of standards for consideration of sex and gender in design and regulation of medical products such as drugs and devices.

Translation of evidence into clinical guidelines and health policy that explicitly considers different populations.

The last activity underpins the implementation pillar and applies to all previous activities:

Monitoring, evaluation, governance and regulation of health and medical research and translation.

Participants repeatedly addressed the need for monitoring and evaluating the impact of interventions such as education and training, and changes to health and medical research, policy and practice. Participants emphasized the importance of a consistent process of review and adaptation over time, based on monitoring and evaluation, and taking account of the dynamic nature of sex- and gender-based research, concepts and terminology. It was also highlighted that this process should go beyond tokenistic metrics to understand and evaluate how institutional change occurs.

A total of 11 outputs were identified as emerging from the activities. These were:

Shared language for discussing sex, gender and intersectional factors;

Resources, training, curricula and advocacy material;

Educated, skilled and aware health research, delivery, policy and governance workforce;

Baseline reporting on current incorporation of sex and gender in health and medical research, policy and practice;

Central hub for networking, engagement and resources.

New or updated policies and guidelines for sex and gender incorporation into health research and practice across the sector;

Comprehensive and inclusive data and evidence on sex, gender and health;

Sex- and gender-informed clinical guidelines, standards, regulations, public policies and strategies;

Reporting on changes in research practice;

Reporting on health indicators across sex and gender domains; and

Established feedback mechanisms for continuous monitoring, evaluation and improvement of health research, policy and practice.

Outcomes reflected the priority activities raised by participants. They strongly felt that the identified activities would lead to improvements in research practice, including (1) improved integration of sex and gender in research design; (2) meaningful, accurate and inclusive data collection and reporting; (3) greater inclusion and participation in health and medical research; and (4) presence, implementation and monitoring of sex and gender research policies in the sector.

Other outcomes reflected the impacts of training, educational and advocacy activities: (5) recognition and application of sex and gender as nuanced, evolving concepts that intersect with other factors that impact health; (6) multidisciplinary sex and gender networks and communities of practice; (7) improved skills, knowledge and understanding of importance of sex and gender in health/medical research, policy and practice; and (8) increased awareness and use of best practice standards and guidelines.

Respondents also identified outcomes that may result from the translation of policies into more appropriate healthcare services and treatment: (9) efficient, inclusive and fit for purpose health interventions and services. This encompassed a variety of possible outcomes raised by stakeholders, including clinicians being able to provide inclusive and appropriate care, better and more cost-effective healthcare delivery, more targeted support for particular populations and more robust medical products. Respondents also felt that the existence of adequate monitoring and evaluation processes would result in an outcome whereby the (10) health sector (is) held accountable for ongoing action to address gender disparities in health outcomes.

Four impacts were identified which fed into the overarching impact of: enhanced health and wellbeing for everyone. Typically, sex and gender research is narrowly viewed as only relevant to women and other marginalized communities, yet the participants emphasized that identifying and evaluating health data benefits all population groups. This was reinforced by the four sub-impacts that were identified: (1) better-quality, nuanced and new health data and information; (2) safe, meaningful and representative participation and experience of diverse groups in health research, policy, practice and care; (3) health and medical sector-wide commitment to reform towards fairer, more inclusive and representative policy and practice; and (4) better and more equitable health and economic outcomes for all.

Barriers and facilitators

Key barriers and facilitators to change were also identified by respondents. These were applicable across the entire ToC map and not just to the achievement of specific outputs, outcomes or impacts.

Two barriers focussed on the influence of entrenched systems and beliefs, including difficulty in changing attitudes and status quo and the influence of discrimination and stigma. Practical barriers included the need or perception of need for more funding, time and resources to meet practical and methodological challenges. Another barrier to change was concern regarding ethics or liability when broadening research inclusivity, particularly when including those who are pregnant or lactating in clinical trials.

Facilitators included societal changes in culture and values that would increase receptivity to change. Another facilitator was leadership, with leadership from organizations and individual champions as well as the equitable gender representation in positions of power across the sector being facilitators for change. Organizations across the sector can also facilitate change through the provision of funding, operational support and expertise and the implementation of accountability measures and mandates.

To the best of our knowledge, this is the first theory of change (ToC) to explicitly outline a common understanding of the sex and gender bias, invisibility and exclusion in health and medical research, policy and practice and outline clear actions and pathways to impact towards enhanced health and wellbeing for all. This work therefore fills an important knowledge-implementation gap in the literature by demonstrating how changes in research policy and practice may create wider impact and the explicit assumptions underlying the guiding future activities and discussion in the field. We identify a range of required actions across evidence generation, translation and implementation that contextualizes the work that many in the sector are already doing, situates the activities of the SGPMR project in this wider context and has the potential to inform the development of future activities. Further, the overarching impact of enhanced health and wellbeing for all is a unifying goal for people working across these sectors, and thus this ToC can be used to reinforce the need to address sex and gender bias, invisibility and exclusion to achieve this impact.

In providing a scaffold for how positive change might occur across medical research, policy and practice through clearly articulated pathways to impact, this ToC also provides important theoretical underpinning to published estimates of macro-level return on investment in gender inclusive research and practice. For example, the donor Women’s Health Access Matters (WHAM) reported that investment of USD$ 300 million in women’s health research across three diseases could result in returns to the economy in excess of USD$ 13 billion by way of improvement in population health and economic productivity [ 48 ]. Assumptions made in the WHAM report regarding how increased investment in research leads to improved health are afforded a more nuanced understanding when considered alongside the pathways to impact articulated in this ToC.

Activities articulated by study participants and represented in the ToC align well with the limited literature that describes initiatives already underway that consider and address disparity in scientific and medical practice. For example, White et al. summarize lessons learnt from funding agencies in developing policies for sex and gender consideration in medical research and identify awareness building, education and collaboration between institutions and continual monitoring and evaluation as necessary to facilitate impact [ 25 ]. Initiatives such as Gendered Innovations and Global Health 50/50 are also actively engaged in building awareness of the need for and value in gender diverse participation in health and medical research and practice and provide guidance to different types of organizations to enhance their practice in this respect whilst monitoring progress against gender inclusion within the global health sector [ 21 , 49 ].

For those who are already working on specific activities such as developing or updating policies and guidelines to impact research practice [ 16 , 50 , 51 ], this ToC can help contextualize this change, inform design and encourage organizations to consider what parallel activities might be needed, such as education and training, consultation with key stakeholders and clarification of concepts across the sector. The ToC demonstrates the importance to those working to address sex and gender issues in evidence, translation and implementation of the need to coordinate their efforts and ensure monitoring and evaluation is communicated to inform practice throughout the pipeline.

Implications for the SGPMR project

The development of this ToC has various implications for the SGPMR project. First, while the ToC spans far beyond than the scope of the project, it supports project affiliates to identify activities to which they can contribute and situate their efforts in a wider change context. Further, as engaging with stakeholders across the health and medical research sector is an activity of the project, this ToC can be used as an advocacy tool to demonstrate the need for change and the role of different organizations in contributing to that change [ 34 , 35 ].

Second, this work also served as a useful activity to reach consensus on the key issues to be addressed and the desired impact of the project. It also facilitated discussion regarding the limitations of this project in achieving long-term impact on health outcomes due to the concentration of activities at the evidence end of the pipeline. Further, this process highlighted the diverse perspectives and priorities of different project stakeholders, related to issues faced by certain populations (namely, cis women, transgender women, gender-diverse people and people with intersex variations), which actions and activities they deemed most important and the areas of the sector they were most familiar with or interested in influencing. This process was beneficial in capturing those different perspectives and working to account for and align the goals of all stakeholders.

Strengths and limitations

A key strength of this work was that it was developed in consultation with stakeholders from various academic and professional backgrounds alongside representatives of communities marginalized because of their sex and gender status. These perspectives have been incorporated in the ToC, enabling an expansive view of sex and gender biases in the sectors impacting diverse groups in different ways and conceptualize how we can create change for the benefit of all.

The development of this ToC has some key limitations. First, as this study was conducted from a sector-wide perspective, it is not centred around a specific intervention and does not trace linear pathways of impact, highlight measurable pre-conditions for success or identify parties responsible for certain actions. Rather, it is a broad conceptual model, reflecting the complexity of the problems and potential solutions, and mapping an array of activities, outputs and potential outcomes. Nevertheless, to our knowledge, this is the first such tool for this sector and therefore has potential to be used across the sector to advance sex- and gender-based policy design, evaluation and impact [ 52 ]. Second, while this ToC covers a broad range of issues, other connected problems, such as the lack of gender equity in the health and medical research, policy and practice workforce were considered out of scope, though others have clearly linked the two issues [ 21 ]. Lastly, we only consulted internal project affiliates for an Australia-based project, and participants were mostly academics, with a small number of end-users working in policy- and community-based organizations. The development of the ToC was based primarily on this consultation, without the benefit of a large literature base, due to the lack of previous research about the efficacy of interventions in this field.

Conclusions

This paper describes the development of a theory of change (ToC) that maps clear pathways to impact for improving the consideration of sex and gender in health and medical research, policy and practice. This ToC is the first of its kind in the field of health and medical research and provides an important theoretical scaffold for institutions and organizations to consider when considering how to actively enhance sex and gender awareness, inclusivity and informed action to contribute to enhancing population health and economic wellbeing.

Availability of data and materials

The datasets used and analysed under study are available from the corresponding author on reasonable request.

Mauvais-Jarvis F, Bairey Merz N, Barnes PJ, Brinton RD, Carrero JJ, DeMeo DL, et al. Sex and gender: modifiers of health, disease, and medicine. Lancet. 2020;396(10250):565–82.

Article   PubMed   PubMed Central   Google Scholar  

Merriman R, Galizia I, Tanaka S, Sheffel A, Buse K, Hawkes S. The gender and geography of publishing: a review of sex/gender reporting and author representation in leading general medical and global health journals. BMJ Glob Health. 2021;6(5): e005672.

Hallam L, Vassallo A, Hallam C, Thompson K, Shanthosh J, Chappell L, et al. Sex and gender reporting in Australian health and medical research publications. Aust N Z J Public Health. 2023;47(1): 100005.

Article   PubMed   Google Scholar  

Geller SE, Koch AR, Roesch P, Filut A, Hallgren E, Carnes M. The more things change, the more they stay the same: a study to evaluate compliance with inclusion and assessment of women and minorities in randomized controlled trials. Acad Med. 2018;93(4):630–5.

Gauci S, Cartledge S, Redfern J, Gallagher R, Huxley R, Lee CMY, et al. Biology, bias, or both? The contribution of sex and gender to the disparity in cardiovascular outcomes between women and men. Curr Atheroscler Rep. 2022;24(9):701–8.

Huded CP, Johnson M, Kravitz K, Menon V, Abdallah M, Gullett TC, et al. 4-Step protocol for disparities in STEMI care and outcomes in women. J Am Coll Cardiol. 2018;71(19):2122–32.

Carcel C, Woodward M, Wang X, Bushnell C, Sandset EC. Sex matters in stroke: a review of recent evidence on the differences between women and men. Front Neuroendocrinol. 2020;59: 100870.

Mogil JS, Chanda ML. The case for the inclusion of female subjects in basic science studies of pain. Pain. 2005;117(1):1–5.

Wang X, Carcel C, Hsu B, Shajahan S, Miller M, Peters S, et al. Differences in the pre-hospital management of women and men with stroke by emergency medical services in New South Wales. Med J Aust. 2022;217(3):143–8.

Farkouh A, Riedl T, Gottardi R, Czejka M, Kautzky-Willer A. Sex-related differences in pharmacokinetics and pharmacodynamics of frequently prescribed drugs: a review of the literature. Adv Ther. 2020;37(2):644–55.

Article   CAS   PubMed   Google Scholar  

Canadian Institutes of Health Research. Gender-Based Analysis Plus (GBA+) at CIHR: Government of Canada; 2020. https://cihr-irsc.gc.ca/e/50968.html . Accessed May 2021.

European Commission. Gender equality in research and innovation: European Commission; 2020. https://ec.europa.eu/info/research-and-innovation/strategy/strategy-2020-2024/democracy-and-citizens-rights/gender-equality-research-and-innovation_en#gender-equality-plans-as-an-eligibility-criterion-in-horizon-europe . Accessed 3 Jun 2021.

Heidari S, Babor TF, De Castro P, Tort S, Curno M. Sex and gender equity in research: rationale for the SAGER guidelines and recommended use. Res Integr Peer Rev. 2016;1(1):2.

National Institutes of Health. NIH guidelines on the inclusion of women and minorities as subjects in clinical research. NIH Guide; 1994.

Office of Research on Women’s Health. Consideration of sex as a biological variable in NIH-funded Research. In: Health NIo, editor. 2016.

Health Canada. Health portfolio sex- and gender-based analysis plus policy: advancing equity, diversity and inclusion Government of Canada; 2022. https://www.canada.ca/en/health-canada/corporate/transparency/corporate-management-reporting/heath-portfolio-sex-gender-based-analysis-policy.html . Accessed 22 Oct 2022

Ludwig S, Oertelt-Prigione S, Kurmeyer C, Gross M, Grüters-Kieslich A, Regitz-Zagrosek V, et al. A successful strategy to integrate sex and gender medicine into a newly developed medical curriculum. J Womens Health. 2015;24(12):996–1005.

Article   Google Scholar  

Mercer-Mapstone L, Bajan S, Banas K, Morphett A, McGrath K. Breaking the binary: conceptions of sex and gender in undergraduate science. Teach Learn Inquiry. 2021. https://doi.org/10.20343/teachlearninqu.9.2.6 .

Muntinga ME, Krajenbrink VQE, Peerdeman SM, Croiset G, Verdonk P. Toward diversity-responsive medical education: taking an intersectionality-based approach to a curriculum evaluation. Adv Health Sci Educ. 2016;21(3):541–59.

Article   CAS   Google Scholar  

Caughey AB, Krist AH, Wolff TA, Barry MJ, Henderson JT, Owens DK, et al. USPSTF to approach addressing sex and gender when making recommendations for clinical preventive services. JAMA. 2021;326(19):1953–61.

Global Health 50/50. Gender Equality: flying blind in a time of crisis, The Global Health 50/50 Report 2021. London, UK; 2021.

Oertelt-Prigione S, Dalibert L, Verdonk P, Stutz EZ, Klinge I. Implementation strategies for gender-sensitive public health practice: a European Workshop. J Womens Health. 2017;26(11):1255–61.

Sex and Gender in Health and Medical Research Australia Group. Project Glossary Australia. 2021. https://www.sexandgenderhealthpolicy.org.au/glossary/ . Accessed Dec 2022.

Woitowich NC, Woodruff TK. Implementation of the NIH sex-inclusion policy: attitudes and opinions of study section members. J Womens Health. 2019;28(1):9–16.

White J, Tannenbaum C, Klinge I, Schiebinger L, Clayton J. The integration of sex and gender considerations into biomedical research: lessons from international funding agencies. J Clin Endocrinol Metab. 2021;106(10):3034–48.

Johnson J, Sharman Z, Vissandjée B, Stewart DE. Does a change in health research funding policy related to the integration of sex and gender have an impact? PLoS ONE. 2014;9(6): e99900.

Haverfield J, Tannenbaum C. A 10-year longitudinal evaluation of science policy interventions to promote sex and gender in health research. Health Res Policy Syst. 2021;19(1):94.

Heidari S, Palmer-Ross A, Goodman T. A systematic review of the sex and gender reporting in COVID-19 clinical trials. Vaccines. 2021;9(11):1322.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Geller SE, Adams MG, Carnes M. Adherence to federal guidelines for reporting of sex and race/ethnicity in clinical trials. J Womens Health. 2006;15(10):1123–31.

Geller SE, Koch A, Pellettieri B, Carnes M. Inclusion, analysis, and reporting of sex and race/ethnicity in clinical trials: have we made progress? J Womens Health. 2011;20(3):315–20.

Peters SAE, Babor TF, Norton RN, Clayton JA, Ovseiko PV, Tannenbaum C, et al. Fifth anniversary of the Sex And Gender Equity in Research (SAGER) guidelines: taking stock and looking ahead. BMJ Glob Health. 2021;6(11): e007853.

Merone L, Tsey K, Russell D, Nagle C. Sex inequalities in medical research: a systematic scoping review of the literature. Women’s Health Rep. 2022;3(1):49–59.

Johnson JL, Greaves L, Repta R. Better science with sex and gender: facilitating the use of a sex and gender-based analysis in health research. Int J Equity Health. 2009;8(1):14.

Sex and Gender in Health and Medical Research Australia Group. Sex and Gender Policies in Health and Medical Research Sydney: Australian Human Rights Institute; 2021. www.sexandgenderhealthpolicy.org.au . Accessed Nov 2021.

The Sex and Gender Sensitive Research Call to Action Group. Sex and gender in health research: updating policy to reflect evidence. Med J Austr. 2020;212(2):57–62.

Hallam L, Vassallo A, Pinho-Gomes A-C, Carcel C, Woodward M. Does journal content in the field of women’s health represent women’s burden of disease? A review of publications in 2010 and 2020. J Womens Health. 2022;31(5):611–9.

Greaves L, Ritz SA. Sex, gender and health: mapping the landscape of research and policy. Int J Environ Res Public Health. 2022;19(5):2563.

Weiss CH. Theory-based evaluation: past, present, and future. New Dir Eval. 1997;1997(76):41–55.

Isabel V. Review of the use of ‘Theory of Change’ in international development. London: UK Department of International Development; 2012.

Google Scholar  

Maini R, Mounier-Jack S, Borghi J. How to and how not to develop a theory of change to evaluate a complex intervention: reflections on an experience in the Democratic Republic of Congo. BMJ Glob Health. 2018;3(1): e000617.

Kathryn S, Lynsay M, Sharon Anne S, Peter C, Janis B, Jane MB, et al. A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. BMJ. 2021;374: n2061.

Department of Foreign Affairs and; Trade Australian Government. Design and monitoring, evaluation and learning standards. Canberra: DFAT; 2023.

AusAID. AusAID’s mulitsectoral contribution to the national response in Papua New Guinea; Annex 10: Theory of Change Canberra: AusAID; 2010.

Weitzman BC, Mijanovich T, Silver D, Brecher C. Finding the impact in a messy intervention: using an integrated design to evaluate a comprehensive citywide health initiative. Am J Eval. 2009;30(4):495–514.

Breuer E, Lee L, De Silva M, Lund C. Using theory of change to design and evaluate public health interventions: a systematic review. Implement Sci. 2016;11(1):63.

Stein D, Valters C. Understanding theory of change in international development. The Asia Foundation; London School of Economics and Political Science 2012.

Australian Bureau of Statistics. Standard for Sex, Gender, Variations of Sex Characteristics and Sexual Orientation Variables ABS Website: Australian Bureau of Statistics; 2021. https://www.abs.gov.au/statistics/standards/standard-sex-gender-variations-sex-characteristics-and-sexual-orientation-variables/2020 . Accessed Jun 2021.

Baird MA, Zaber AZ, Chen A, Dick AW, Bird CE, Waymouth M, et al. The case to fund women’s health research: an economic and societal impact analysis. Santa Monica: RAND; 2021.

Schiebinger L, Klinge, I, Sánchez de Madariaga, I, Paik, HY., Schraudner, M, and Stefanick, M. Gendered Innovations in Science, Health & Medicine, Engineering and Environment Stanford: Stanford University; 2011–2021. https://genderedinnovations.stanford.edu . Accessed Jun 2021.

Hunt L, Nielsen MW, Schiebinger L. A framework for sex, gender, and diversity analysis in research. Science. 2022;377(6614):1492–5.

Association of Australian Medical Research Institutes. AAMRI gender equity diversity and inclusion strategy and action plan. Melbourne: AAMRI; 2021.

LaFond A, Cherney M. A theory of change for guiding the integration of human-centered design into global health programming. Glob Health Sci Pract. 2021;9(Suppl 2):S209–16.

Download references

Acknowledgements

The Theory of Change figure was designed with help from Deborah Bordeos.

All authors’ work on this study was funded from an anonymous philanthropic grant. The funder had no role in the design, execution, analysis or write-up of this study.

Author information

Thomas Gadsden and Laura Hallam contributed equally and hold joint first authorship.

Authors and Affiliations

The George Institute for Global Health, University of New South Wales, Sydney, Australia

Thomas Gadsden, Laura Hallam, Cheryl Carcel, Robyn Norton, Mark Woodward & Laura E. Downey

Australian Human Rights Institute, University of New South Wales, Sydney, Australia

Laura Hallam

The George Institute for Global Health, Imperial College London, London, United Kingdom

Robyn Norton, Mark Woodward & Laura E. Downey

The Human Rights Institute, University of New South Wales, Sydney, Australia

Louise Chappell

You can also search for this author in PubMed   Google Scholar

Contributions

L.H., T.G., C.C., M.W., R.N. and L.D. conceptualized the study design. L.H., T.G. and L.D. collected and analysed all data, and wrote the first draft of the manuscript. C.C., R.N., M.W. and L.C. reviewed the draft manuscript and provided scientific and editorial insights to strengthening the manuscript.

Corresponding author

Correspondence to Laura E. Downey .

Ethics declarations

Ethics approval and consent to participate.

This study received ethical approval from the University of New South Wales (HC220443). All participants provided written consent to participate in the study.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., supplementary material 3., supplementary material 4., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Gadsden, T., Hallam, L., Carcel, C. et al. Theory of change for addressing sex and gender bias, invisibility and exclusion in Australian health and medical research, policy and practice. Health Res Policy Sys 22 , 86 (2024). https://doi.org/10.1186/s12961-024-01173-z

Download citation

Received : 14 August 2023

Accepted : 25 June 2024

Published : 15 July 2024

DOI : https://doi.org/10.1186/s12961-024-01173-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Sex and gender
• Medical research
• Health policy
• Evidence-based practice

Health Research Policy and Systems

ISSN: 1478-4505

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]