• Facebook Social Icon
  • Twitter Social Icon

A Meta-Study of Democratic Deliberation: Advancing Theory and Practice

This three-year Australian Research Council funded project began in 2018

Project Summary

The project combines a meta-study and comparative case study to develop a leading edge understanding of political deliberation by analysing and synthesising results from available studies of deliberation. It aims to reconcile conflicting findings and provide the first comprehensive, theoretically-grounded account of defensible claims about political deliberation. The project will compile the source material and findings in a publicly-available database to facilitate standardisation and enhancement of future research in the field. It will seek to settle important questions that remain among deliberative democrats and, more practically, facilitate avenues for democratic reform in an area where the need for renewal is increasingly pressing.

Background information

Project Aims

The project will assemble a strong, coherent team, with strong international networks, to perform four important functions:

  1. Conceptual verification and clarification. Updating Advancing the theory of deliberation based on evidence will combine quantitative meta-analysis and intensive small n qualitative-interpretative comparative case study research. The aim is to develop a leading edge understanding of political deliberation by comparing results from available studies making a claim to the analysis of deliberation. It will seek to reconcile conflicting findings in the field by determining which variables account for the differences in claims that are made, and whether the solution requires reconceptualization. This study will be the first systematic and comprehensive account of what claims really can be made about the nature of political deliberation in different settings and contexts. The ultimate aim is to hold accountable those who claim to have observed ‘deliberation’, and help those who seek to design to achieve it.

  2. Enhancing future research capacity. The project will set the standard for future empirical research in the field, as well as increasing capacity. It will compile all the available material — studies, variables, survey instruments, data and findings building on a database to host findings, resources (e.g. survey items) and design features from existing research on micro-deliberation, and provide this as resource that is available to all researchers in the field as mechanisms for standardizing and enhancing future research. The database will be linked to Participedia (Fung and Warren, 2011) (https://www.participedia.net) to provide open access, building on existing relationships with the University of Canberra based Centre for Deliberative Democracy and Global Governance (hereon, The Centre). Participedia is an open source repository of information about participatory processes around the world — this project will enhance the research potential of that facility.

  3. Update theories of deliberative democratisation in the Anthropocene. Dryzek and Niemeyer will combine their considerable experience on the question of deliberative environmental governance, using this project to focus on the dimension of deliberation on environmental issues, particularly climate change, and assess how well claims are backed by evidence.

  4. Establishing design parameters for deliberative practice and institutionalisation. In terms of practice, the aim of this project is to develop an authoritative and actionable account of what makes a ‘good’ deliberative forum, and what accounts for the differences in outcomes. By building on the Centre’s network of deliberative practitioners, we hope to provide responses to the questions our Centre often gets asked: What’s a good design of deliberative forum for this particular issue? The project and its findings will also be used in conjunction with The Centre’s increased move into the spheres of practitioner, public and decision maker in the form of the Democratic Innovation Initiative, being developed in cooperation with the Museum of Australian Democracy, which seeks to inform innovation in democratic practice (IGPA, 2016).

 

Background

Deliberative theory needs to advance foundational questions in terms of what deliberation is (process), what it does (outcomes), how these two inter-relate, and how deliberation is achieved (design). Debates regarding deliberative process primarily concern the manner in which deliberation best proceeds (role of rhetoric and story telling, for example Dryzek, 2010), versus more formal Habermasian public reasoning (see Bächtiger, Niemeyer et al., 2010). However, the comparative effectiveness of different types of deliberative design and the role of other factors has not been tested. Nor has the theory of deliberation been properly updated to take into account the dimension of how individuals “listen” to arguments (Dobson, 2014), or the effect of dispositional state (or deliberative stance Owen and Smith, 2015), and situation and personal characteristics in shaping how deliberation proceeds (Jennstål 2016).

At its core deliberation is supposed to involve changing opinions (or preference transformation) as individuals encounter, evaluate and absorb alternative arguments (Barabas, 2004; Mackie, 2006). However, the outright necessary deliberativeness of preference or opinion transformation has increasingly become questioned (e.g. Baccaro, Bächtiger et al., 2016), and that this operationalization of deliberation has led to incomplete, or even misleading conclusions (see Niemeyer, 2017b).

Niemeyer and Dryzek (2007) have previously reconceptualised the ideal ‘ends of deliberation’ in terms of metaconsensus (see also Dryzek and Niemeyer, 2006). However, until recently, an approach for measuring metaconsensus has not been available. A promising solution, in the form of ‘integrative metaconsensus,’ has been under development by Niemeyer as a test to how well deliberative outcomes are intersubjectively responsive to all relevant reasons — as opposed to emphasis of symbolic issues, as induced by populism (see Niemeyer, 2011). Methodological challenges to operationalizing this measure have now been overcome as part of Niemeyer’s recent Future Fellowship (FT110100871), with a well-developed metric based on concordance between measured subjective values and beliefs (reasons) and policy preferences (choices) exhibiting dramatic improvement across all case studies, except control groups (Niemeyer, 2017a). More promisingly, this metric can be used to model the effect of deliberative designs. When features of deliberative design (exercises to activate deliberative norms, reliance on voting, group size, process length and case study type) are modelled in a regression, strongly significant effects emerge, where ‘norm activation’ (via this type of group exercise) in particular is strongly positive and reliance on voting is strongly negative in terms of deliberative effect as measured by integrative metaconsensus.

These preliminary results will be expanded in this research, which will also develop easier to obtain proxies for integrative metaconsensus by comparing case study results using different metrics to those in Niemeyer’s database of studies. A selection of benefits (dependent variables) ascribed to deliberation and tested in experimental settings, such as improved knowledge (Grönlund, Setälä et al., 2010; Luskin, Fishkin et al., 2002) and increased internal political efficacy (Morrell, 2005) will be tested to examine whether they occur in parallel to each other or are individually a function of particular design features.

The process and outcome of deliberation also needs to be tested against other dimensions, such as the nature of the topic. For example, Niemeyer identifies climate change as a particularly difficult issue for achieving deliberation (Niemeyer, 2017a). Given his and Dryzek’s focus past work on environmental governance, this will be of particular focus in the project to address the question of deliberative governance in the Anthropocene and the conditions where deliberation is possible in respect to complex environmental issues (Niemeyer, 2014b), critically assessing and updating many claims made in the literature from Dryzek (1987) to Baber & Bartlett (2015).

The project will also critically assess institutional features from a deliberative perspective, including that of voting, which has come under increased scrutiny in democratic theory (Reybrouck and Waters, 2016). Niemeyer, Felicetti et al. (2016) have observed the distorting effect of voting as a stopping or decision rule on the deliberative process. So too have optimistic claims about online deliberation (Strandberg and Grönlund, 2014), which are in serious need of careful, systematic scrutiny and will be included as a design feature explored in this study.

Another under-appreciated factor influencing both the possibilities for deliberation, as well as the uptake of its outcomes, is that of institutional and cultural context. In comparison to many studies that draw assertions based on single, geographically specific studies, there is a compelling argument that the background settings interact with deliberative designs in ways that impact on the deliberative process and outcomes (Felicetti, Niemeyer et al., 2016). Jennstål (2016) — who is also a candidate to join the project — demonstrates this is also the case where context and personality interact in respect to willingness to participate in deliberation. However, to date, there is no systematic assessment of these effects. This project will address this lacuna.

Informing Deliberation in Practice

There has been an explosion of deliberative forums that seek to give voice to citizens in collective decision-making. These ‘minipublics’, ranging varying in design from citizen juries to deliberative polls, and ranging in scale from local town meetings to transnational citizen consultations, have rightly or wrongly been valorised as the paragon of democratic innovations that can rectify the deficits of contemporary politics (Fung, 2003; Grönlund, Bächtiger and Setälä, 2014). These forums have become so popular, such that there is a growing industry of deliberative consultants and ‘proprietary participatory products’ that claim to maximise ‘democratic goods’ (Hendriks and Carson 2008). Others take this a step further. In the book Against Elections: A Case for Democracy, for example, David Van Reybrouck (2016) calls for sortition – the installation of a random sample of citizens who will deliberate on policies in place of elected politicians.

Innovative thinking is essential in responding to crisis levels of democratic deficit. But even the most intuitively appealing prescriptions require careful and critical scrutiny. They also need the best possible theoretical framework and empirical evidence as their basis. As the ‘deliberative turn’ in democratic theory enters its twenty-fifth year, it is crucial to take stock of theory, evidence, and the grand claims made by both deliberative scholars and practitioners.

In an earlier Australian Research Council (ARC) funded study (DP0558573) John Dryzek and Simon Niemeyer set about clearing up important issues concerning the micropolitics of deliberation. By combining empirical observation and theoretical updating, they made groundbreaking contributions to foundational questions concerning the status of consensus (Dryzek and Niemeyer, 2006, published in American Journal of Political Science) and selection procedures to achieve discursive representation (Dryzek and Niemeyer, 2008, published in the American Political Science Review). In light of the growth in claims about democratic innovations, and as deliberative democrats embark on the important task of accounting for the operation of complex democratic systems from a deliberative perspective (e.g. Niemeyer, Curato & Bächtiger, 2016), an even larger, more sophisticated intervention is necessary. This project will provide critically needed clarity in terms of what deliberation is and how it can be observed, the settings under which it is achieved, its democratic benefits, as well as the practical and institutionalisation possibilities.

Important questions remain among deliberative democrats: what constitutes ‘good’ democratic innovations; how design features impact on deliberative process and outcomes; the nature of collective problems they can productively be applied to, including complex global environmental issues; and whether they are cost effective? . These questions continue to be unanswered because a lot of work remains to be done in establishing a clear and definitive link between deliberative theory, empirical research, and democratic practice (see Boswell, Niemeyer et al., 2013). In terms of theory, the problem begins at the definitional stage. Authors such as Diana Mutz (2008, p.525) have lamented the diversity of definitions of deliberation. Although some diversity can be productive, both theory and practice has been held back, to some extent, by excessive ‘concept stretching’ (Steiner, 2008). In terms of methodology, the literature has, for the most part, been shaped by small scale experiments, micro-analyses, and in-depth case studies that make grand claims about the benefits of deliberation (social learning, open-mindedness, civic virtues) and their pathologies (group polarisation, inequalities in voice), without a definitive understanding of the cause and effect of institutional design or the confounding role of context and culture. Therefore, deliberative democracy’s promise is held back by lack of clarity, misinterpretation, or poorly justified claims.

This project will provide the critical tools for addressing these issues. To imagine the future of deliberative politics, we first require an objective assessment of the promise public deliberation holds and the mechanisms whereby it is achieved. We need a better, more integrative, problem-driven approach to social science that informs theory and speaks to real world problems (Warren, 2017), resulting in theory and practice that is both universal and sensitive to context. And when a theoretician, empirical researcher or practitioner asserts that one has observed ‘successful deliberation,’ we urgently need a clear yardstick to test this claim.

Investigators

The research team is composed of two senior academics (Dryzek and Warren) who are considered to be the pillars of deliberative democratic theory, two mid-career academics (Niemeyer and Bächtiger) who have been at the forefront of methodological innovations in the study of deliberative practice and one early career academic (Curato) who has established a track record of studying deliberative forums and their role in broader political systems. The team is purposefully kept small to maintain cohesiveness and a clear division of labour, while engaging with a wider community of international scholars.

The figure below summarises the responsibilities of each investigator indicated by initials. All investigators have experience working together on co-authored publications and collaborative projects—a number of them funded by the Australian Research Council (ARC).

 

In terms of specific roles, Simon Niemeyer (CI) will take the role of lead CI in the project. Niemeyer recently concluded his Future Fellowship (FT110100871) and will be involved in most aspects of the project, including shouldering the main administrative responsibilities. Specifically he will oversee the literature review (1a) and assist with development of the sampling frame (2). He will then finalise the assessment methods (1b), help supervise the collection of data (3a) and coding of case studies (3b). He will oversee part of the initial analysis (4, 5a) and apply his experience in linking these results to the more intensive case study comparisons (5b). And, in terms of outputs, he will contribute to the theory and practice article in Year 1 (output A), the special issue following the first symposium (B), the final manuscript on design and practice (F) as well as a dedicated practitioner sourcebook (G). In particular, he will partner with Dryzek to author a manuscript translating the evidence gained from the project to the question of deliberative governance of difficult, global environmental issues such as climate change (H), as well as the book updating theoretical findings (I).

John Dryzek (CI) is a Laureate Fellow (FL140100154, ending December 2019). He will reprise his role in partnership with Niemeyer on their original research in micropolitics (DP0558573). This will mainly take the form of devoting his formidable skills at the conclusion of his Laureate to translating empirical findings into important theoretical insights, and contribute to the development of output K, dedicated journal articles (H-J) as well as the environmental governance volume with Niemeyer (G). During the first two years of the project his more limited involvement will include participation in the literature review and preparation of a state of theory article (A) and symposium articles (D).

Nicole Curato (CI) is a Discovery Early Career Research Award Fellow (DE150101866) and will be in -charge of data collection and interpretive comparative case studies of selected deliberative forums (3a, 5b). She will take a leadership role in the project’s communications and dissemination strategy, given her extensive experience in engaging broadcast and digital media to disseminate research findings. She will also take a leadership role in drafting the sourcebook for practitioners (L), and editing the special issue journal journal symposium (D) and compilation of the book compendium (F), as well as write a book proposal related to the project (K). Her DECRA ends in 2018 in time for this project’s commencement. She will spend 50% of her time on this project.

Andre Bächtiger’s (PI) expertise in empirical methods will be critical to the analytical stages of the project (1a, 2, 1b, 4,5a, 5c) and in producing scholarly publications (A, D, F, H-J, K). His teaching commitments in Stuttgart will be a limiting factor for his participation early in the project, but this will be facilitated by travel to Germany by Canberra-based staff (Niemeyer, Curato) and the PhD scholar whom he will co-supervise as part of the panel during the early stages of the project, as well as the application for an International Collaboration Award.

Mark Warren’s (PI) involvement in the project will primarily involve overseeing data collection in Participedia (in conjunction with Curato) and linking the Participedia data with the project’s database (3a). Warren is currently working on a Social Sciences and Humanities Research Council of Canada-funded project on Participedia, of which the Centre is a part by providing information on Australian case studies. His involvement with this DP will take up 10% of his time during years one and two. In year three, his eminence as a leading scholar in normative deliberative theory will be deployed during the preparation of the theoretical manuscript (K).

While the investigators of this project are limited to five academics, this research taps on a broader network of empirical scholars of deliberative democracy who will attend the symposia, provide feedback on the research process and extend advice on the project’s trajectory. Included in the project’s wider team include are both established and emerging scholars, such as Kimmo Grönlund from Åbo Akademi (Finland), Maija Setälä from the University of Turku (Finland), Julia Jennstål from Uppsala University (Sweden), and Sofie Marien from KU Leuven (Belgium). It will also be expanded to other scholars such as James Fishkin from Stanford (United States), John Gastil, Brigitte Geißel (Goethe-Universität Frankfurt) and Tali Mendelberg from Princeton University (United States). 

Project Quality and Innovation

Many claims about the nature and effects of deliberation are based on single case studies, or at best a collection of disparate studies, most lacking rigorous control (Setälä and Herne, 2014). There has been a call to conduct more experimental research that seeks to isolate particular effects (Mutz, 2006; Karpowitz and Mendelberg, 2011). However, such calls are insensitive to the potential importance of context, sampling, and interaction effects on deliberation, where different study populations produce different dynamics; or that deliberation (however defined) might exhibit emergent properties — where interactions mean the deliberation is more than the sum of its parts — that are easily confounded in ways that cannot be controlled by a single, geographically specific experiment (Böker, 2016). Other potentially confounding factors include the nature of the issue in question, where the wider effects of issue complexity and politicisation influence the pre-deliberative settings. Climate change is a standout example of an issue where achieving deliberation may be difficult, perhaps dependent on location. As such, the topic of climate change will be a particular focus for the research, given that it also represents one of the greatest challenges to democratic governance (Niemeyer, 2014b).

Dealing with this complexity and the limitations of pure experimentation on deliberation requires a different approach. Meta-studies, common in the natural sciences, are relatively rare in the social sciences outside psychology. Fortunately, there now exists a considerablythere are a large number of minipublic ‘experiments’ — numbering well into the hundreds — from which systematic conclusions might be drawn using methods involving meta-analysis. Meta-analysis is usefully deployed where a field has experienced a huge growth in empirical research, without standardisation of methods, concepts or design. It permits the collection of data across case studies and systematic examination of effect size, heterogeneity of effect and design features that help explain this heterogeneity (Sánchez-Meca and Marín-Martínez, 2010). But this will only be part of the approach. The project will also involve careful comparison of studies using interpretive methods, and combining the results to test and, where necessary, update deliberative theory. The study will therefore be innovative in approach, and demonstrate how meta-studies should be recognised and used in the social sciences.

 To our best knowledge there are only couple of examples of systematic meta-analysis conducted on deliberation to date — although there exist a number of reviews of empirical evidence in the field of deliberative democracy (Delli Carpini, Cook et al., 2004; Mendelberg, 2002; Thompson, 2008). These include a study by Marina Lindell as part of her PhD research, and current research at Goethe-University Frankfurt (e.g. Hess, Brehme et al., 2015). Lindell’s (2011) approach involveds categorising six types of effects of deliberation (opinion change, absence of group polarisation, knowledge gain, internal efficacy, external efficacy, increased political participation, and deliberative capacity) and testing the effect of independent variables pertaining to deliberative design (plenary sessions, group discussion, rotation among tables, facilitation, sequencing of stages of deliberation), manner of information provision (background info, expert witness, stakeholder witness, site visit, participant questions, and information requests), manner of decision making (common statement, consensus, voting, other) and connection to policy process (decision maker commissioned, neutral organiser, contract, impact, and parallel activities). The study used a ‘method of agreement’ (Caramani, 2009) approach using Boolean coding to test whether the presence of an independent variable had an effect on one or more of the dependent variables across the cases where information on that variable was available (Schneider and Wagemann, 2013). Given her experience on this topic, Lindell is slated to be part of the team as a research associate.

 

Despite being limited to only nineteen case studies (comprising Citizens’ Juries, Deliberative Polls, Citizens’ Assemblies and dedicated experiments) — with a good deal of missing data — the study did produce a number of relevant findings. In particular, contrary to the common assertion that minipublic deliberation should be decisive and directly plugged into decision making, lest participants become de-motivated — decisiveness had no impact of deliberative quality (Lindell, 2011). This supports the conclusion that minipublic deliberation might be better deployed in deliberative innovation than as a decision making mechanism (Niemeyer and Jennstål, Forthcoming).

Methodology

This study will go well beyond this It will involve a much largeapproach, in terms of sample size — beyond anything done in the field so far —, methodological sophistication, wider range of data, more detailed coding of variables; as well as deploying an innovative combination of quantitative and interpretive methods and linkage to theory and practice. As such, it will produce more definitive findings with strong explication of cause and effect, covering a wider range of issues. And it will have a strong impact on theory, practice and ideas about prospects for democratic renewal.

The overall research process is illustrated in the figure above. Broadly, the stages follow those ascribed by standard meta-analytic methodology: (1) Defining the research question; (2) literature search; (3) coding of studies; (4) calculating an effect-size index; (5) statistical analysis and interpretation, and (6) publication (Sánchez-Meca and Marín-Martínez, 2010) — where the interpretive phase will deploy more intenstive methods of comparative case study analysis.

Stages 1 and 2 will be interactive. The analytical design (variables) will take place in partnership with an international network of researchers (see above) who will be gathered at a symposium in Canberra in conjunction with the Centre’s growing Deliberative Democracy Summer School. Verification of the sampling model will be followed by data collection and entry into the database (3a) and coding (3b). Calculation of effect size index — making reported effects across studies comparable — and analysis will then be conducted (4, 5a), the results of which will inform the comparative case study analysis (5b), which in-turn will inform the updated meta-analysis (5c).

The meta-study will draw from and update the existing range of methodology the investigators used in their study of deliberative forums. All investigators have experience combining quantitative and qualitative approaches to studying deliberative procedures, as evidenced by their publication records.

The study will sample a set of case studies, numbering at least (n >100 — in addition to Niemeyer’s existing database of fourteen of his studies and five others —, with final numbers depending on ease of acquiring data, most of which will involve direct contact with researchers; 20 of which will be dedicated to the topic of climate change. These will be ; drawn from among the nearly 700 listed in Participedia) across a wider range of deliberative sites, including online deliberation and (where data is available) real world deliberation. The selection of both dependent and independent variables will draw from more recent developments in the field of deliberative democracy. It will also include the dependent variable of integrative metaconsensus as one of the outcomes. Although data for integrative metaconsensus is only available for case studies collected by Niemeyer, the study will pool data for other dependent variables to test whether proxy measures that are more easily obtainable can be used. To the extent this is the case, this will comprise an important methodological innovation that will improve and potentially help standardise the assessment of deliberative outcomes.

This approach, as well as connecting elements of each case study together using a relational database, will help to overcome a potential weakness of the meta-analytic approach in respect to deliberation. Most important of these involves the conceptual uncertainty in respect to what deliberation does and what elements comprise an effect, a contributing factor (independent contextual variable), a process variable, or a background condition.

While powerful, the results of the meta-analytical study will still require careful interpretation and, where necessary, updating of the analytical framework. Certain questions that lend themselves to clear hypotheses will be addressed by meta-analysis, but a good deal of work will remain in terms of interpreting results and working through the implications for both theory and practice. Here mixed methods will be deployed. The results of quantitative analysis will be used to inform both the selection and investigation of studies using more detailed interpretative analysis. Two broad approaches will involve examining cases that appear similar, but produce different outcomes to ascertain whether the differences are due to random effects, research design issues or whether the theoretical frame used for analysis is inadequate. The second strategy involves different case studies that converge on similar effects. This will inform whether theory is adequate or, more interestingly, whether deliberation is an emergent property, with the theoretical and practical implications that are associated. If the model of deliberation does need updating, which is almost certain, then this will be tested in the second round of meta-analysis.

Benefit

The project will yield enormous benefits that extend beyond immediate academic interest. In addressing critical, core questions, facilitating conceptual clarity and defining the causal links in deliberation and its benefits, it will break the deadlock in deliberative democracy. This clarity will yield wider benefits for the larger project of evaluating whole democratic systems from a deliberative perspective and inform routes for improved democratisation in troubled times (Felicetti, Niemeyer et al., 2016; Niemeyer, Curato et al., 2016; Niemeyer and Jennstål, Forthcoming).

The project will also provide a basis for increasing outreach into practitioner and policy making communities, providing the source material for engagement activities associated with the Democratic Innovation Initiative (IGPA 2016). The benefit of identifying clear mechanisms for better linking citizens to decision making, improving public debate, building deliberative capacity and producing improved political outcomes over the longer term far outweigh the costs involved. This includes the environmental/climate change governance component that will comprise much of Niemeyer’s and Dryzek’s focus, which they can use to build on foundations that have been laid by previous research.

References

Baccaro, L., A. Bächtiger and M. Deville (2016). "Small Differences that Matter: The Impact of Discussion Modalities on Deliberative outcomes." British Journal of Political Science 46(3): 551-566.

Bächtiger, A., S. Niemeyer, M. Neblo, M. R. Steenbergen and J. Steiner (2010). "Disentangling Diversity in Deliberative Democracy: Competing Theories, Their Empirical Blind-spots, and Complementarities." Journal of Political Philosophy 18(1): 32–63.

Barabas, J. (2004). "How deliberation affects policy opinions." American Political Science Review 98(4): 687-701.

Böker, M. (2016). "Justification, critique and deliberative legitimacy: The limits of mini-publics." Contemporary Political Theory.

Boswell, J., S. J. Niemeyer and C. M. Hendricks (2013). "Julia Gillard's Citizens' Assembly Proposal for Australia: A Deliberative Democratic Analysis." Australian Journal of Political Science 48(2): 164–178.

Caramani, D. (2009). Introduction to the comparative method with Boolean algebra. Los Angeles, SAGE.

Delli Carpini, M. X., F. L. Cook and L. R. Jacobs (2004). "Public Deliberation, Discoursive Participation, and Citizen Engagament: A Review of the Empirical Literature." Annual Review of Political Science 7: 315-344.

Dobson, A. (2014). Listening for Democracy: Recognition, Representation, Reconciliation. Oxford, Oxford University Press.

Dryzek, J. S. (1987). Rational Ecology: Environment and Political Ecology. New York, Basil Blackwell.

Dryzek, J. S. (2010). "Rhetoric in democracy: A systematic appreciation." Political Theory 38(3): 319–339.

Dryzek, J. S. and S. J. Niemeyer (2006). "Reconciling pluralism and consensus as political ideals." American Journal of Political Science 50(3): 634–649.

Dryzek, J. S. and S. J. Niemeyer (2008). "Discursive Representation." American Political Science Review 102(4): 481–494.

Evans, M., M. Halupka and G. Stocker (2016). Who do you trust to run the country? Democracy, trust and politics in Australia. Canberra, Institute for Governance and Policy Analysis.

Felicetti, A., S. Niemeyer and N. Curato (2016). "Improving deliberative participation: connecting mini-publics to deliberative systems." European Political Science Review 8(3): 427-448.

Fung, A. and M. E. Warren (2011). "The Participedia Project: An Introduction." International Public Management Journal 14(3): 341-362.

Grönlund, K., M. Setälä and K. Herne (2010). "Deliberation and civic virtue: lessons from a citizen deliberation experiment." European Political Science Review 2(1): 95–117.

Hendriks, C. M., & Carson, L. (2008). Can the market help the forum? Negotiating the commercialization of deliberative democracy. Policy Sciences, 41(4), 293.

Hess, P., M. Brehme and B. Geissel (2015). Examining Deliberation – Meta-Analysis of Case Studies. ECPR General Conference. Montreal.

IGPA (2016). Three Big Ideas, Institute for Governance and Policy Analysis.

Jennstål, J. (2016). Deliberative Participation and Personality: The Effect of Traits, Situations and Motivation. Political Behavior, Centre for Deliberative Demcracy and Global Governance, Institute for Governance and Policy Analysis, University of Canberra.

Kaarbo, J. and R. K. Beasley (1999). "A Practical Guide to the Comparative Case Study Method in Political Psychology." Political Psychology 20(2): 369–391.

Karpowitz, C. and T. Mendelberg (2011). An experimental approach to citizen deliberation. Cambridge Handbook of Experimental Political Science. J. N. Druckman, D. P. Green, J. H. Kuklinski and A. Lupia. Cambridge, Cambridge University Press: 258–272.

Lindell, M. (2011). Same but Different - Similarities and Differences in the Implementation of Deliberative Mini-publics. ECPR General Conference. University of Reykjavik, Iceland.

Luskin, R. C., J. S. Fishkin and R. Jowell (2002). "Considered Opinions: Deliberative Polling in Britain." British Journal of Political Science 32(3): 455–487.

Mackie, G. (2006). "Does democratic deliberation change minds?" Politics, Philosophy & Economics 5(3): 279–303.

Mendelberg, T. (2002). The deliberative citizen: theory and evidence. Research in Micropolitics: Political Decisionmaking, Deliberation and Participation. M. X. Delli Carpini, L. Huddy and R. Shapiro. Greenwhich, Conn., JAI Press. 6: 151–193.

Morrell, M. E. (2005). "Deliberation, democratic decision-making and internal political efficacy." Political Behavior 27(1): 49-69.

Mutz, D. C. (2006). Hearing the other side: deliberative versus participatory democracy. New York, Cambridge University Press.

Mutz, D. C. (2008). "Is deliberative democracy a falsifiable theory?" Annual Review of Political Science 11: 521–538.

Niemeyer, S. J. (2011). Intersubjective Reasoning and the Formation of Metaconsensus. Technologies for Supporting Reasoning Communities and Collaborative Decision Making: Cooperative Approaches. J. Yearwood and A. Stranieri. Hershey, Penn., IGI Global: 18–37.

Niemeyer, S. J. (2014a). Scaling Up Deliberation to Mass Publics: Harnessing Minipublics in a Deliberative System. Deliberative Mini-Publics: Practices, promises, pitfalls. K. Grönlund, A. Bächtiger and M. Setälä. Essex, ECPR Press.

Niemeyer, S. J. (2014b. "A defence of (deliberative) democracy in the Anthropocene." Ethical Perspectives 21(1): 15–45.

Niemeyer, S. J. (2017a). Deliberation and Integrative Metaconsensus. Centre for Deliberative Democracy and Global Governance working paper series, Centre for Deliberative Demcracy and Global Governance, Institute for Governance and Policy Analysis, University of Canberra.

Niemeyer, S. J. (2017b). When does deliberation really begin? It depends on how you measure it. Centre for Deliberative Democracy and Global Governance working paper series, Centre for Deliberative Demcracy and Global Governance, Institute for Governance and Policy Analysis, University of Canberra.

Niemeyer, S. J., N. Curato and A. Bächtiger (2016). Assessing the deliberative capacity of democratic polities and the factors that contribute to it. ECPR Joimt Sessions. Pisa.

Niemeyer, S. J. and J. S. Dryzek (2007). "The Ends of Deliberation: Metaconsensus and Intersubjective Rationality as Deliberative Ideals." Swiss Political Science Review 13(4): 497–526.

Niemeyer, S. J., A. Felicetti and N. Curato (2016). Systemic Assessment of Deliberative Assemblies: A Comparative Approach. The 112th American Political Science Association Annual Meeting,  2016, Philadelphia, PA.

Niemeyer, S. J. and J. Jennstål (Forthcoming). From Minipublics to Deliberative Democracy: Scaling Up Deliberativeness and Subverting Political Manipulation. Handbook of Deliberative Democracy. A. Bächtiger, J. S. Dryzek, M. E. Warren and J. J. Mansbridge. Oxford, Oxford University Press.

Owen, D. and G. Smith (2015). "Survey Article: Deliberation, Democracy, and the Systemic Turn." Journal of Political Philosophy 23(2): 213–234.

Van Reybrouck, D. v. and L. Waters (2016). Against elections: the case for democracy. London, The Bodley Head.

Sánchez-Meca, J. and F. Marín-Martínez (2010). "Meta-analysis in Psychological Research." International Journal of Psychological Research 3(1): 151–163.

Schneider, C. Q. and C. Wagemann (2013). "Doing Justice to Logical Remainders in QCA: Moving Beyond the Standard Analysis." Political Research Quarterly 66(1): 211-220.

Setälä, M. and K. Herne (2014). Normative Theory and Experimental Research in the Study of Deliberative Mini-Publics. Deliberative Mini-Publics: Practices, promises, pitfalls. K. Grönlund, A. Bächtiger and M. Setälä. Essex, ECPR Press: 59–72.

Steiner, J. (2008). "Concept Stretching: the Case of Deliberation." European Political Science Review 7(2): 186-190.

Strandberg, K. and K. Grönlund (2014). Online Deliberation: Theory and Practice in Virtual Mini-Publics. Deliberative Mini-Publics: Practices, promises, pitfalls. K. Grönlund, A. Bächtiger and M. Setälä. Essex, ECPR Press: 93–108.

Thompson, D. F. (2008). "Deliberative democratic theory and empirical political science." Annual Review of Political Science 11: 497–520.

Warren, M. E. (2017). "A Problem-Based Approach to Democratic Theory." American Political Science Review 111(1).