When Is Deliberation Useful for Optimism Governance? Experimental Evidence from Retro Funding 4’s Deliberative Process
→ Full draft of the academic paper (linked here soon)
Summary
One of the most important questions for Optimism governance is: How do we increase the quality of discussion and information exchange so that participants can make informed decisions? One idea is to facilitate spaces dedicated to deliberation, and deliberative processes are gaining traction as a democratic tool for both offline (eg, Ireland’s citizens assembly) and online (eg, Meta’s community forums) governance. Despite the rise in adoption, there are open questions about when and how deliberation is useful. Moreover, web3 introduces new opportunities to study the effects of deliberation in a governance structure that progressively decentralizes over time, and in the case of Optimism participants hold immediate decision-making power and the results of the deliberative process were binding. To understand how deliberation might be useful in our governance system, we designed a deliberative process experiment prior to Retro Funding 4 in Optimism’s Citizens House. (See these links for more context on the topic and logistical details of Optimism’s deliberative process.)
In our experiment, we find that:
(1) Participants in the deliberative process significantly increased their knowledge on arguments for and against the contested topic
(2) Participants in the deliberative process significantly increased their trust in the opinions of other badgeholders
(3) Deliberation seemed to increase support for deliberation as an effective decision-making tool
(4) Deliberation moved participants toward consensus on only one of four categories
(1) Deliberation increases knowledge on arguments for and against contested topic
- Theory: Potential benefits of deliberation include more informed and reflective judgements, a greater sense of internal political efficacy (that is, confidence people have in their personal knowledge and competence to engage in political discussions and make informed decisions).
Outcome to test Hypothesis 1: Deliberation Increases Knowledge
| Outcome: | How would you rate your knowledge on the arguments for/against deducting external funding (e.g., VC funding, Optimism grants, or other grants) from public goods rewards? |
| Scale: | 0 (very low knowledge) ... 4 ... 7 (very high) |
i. Participants had higher post-deliberation knowledge than pre-deliberation
| Paired T-test Results | |
|---|---|
| MeanPre-deliberation | 4.96 |
| MeanPost-deliberation | 5.83 |
| Mean Difference | +0.88*** |
| Standard Error | (0.11) |
| t-statistic | 3.6 |
| Degrees of Freedom | 23 |
Comparing pre/ post-survey responses, participants increased about 0.88 on a 7-point scale measuring their knowledge on the deliberated topic. With a p-value of 0.003 this is a statistically significant shift in opinions among participants, and substantively moving nearly 1 point on a 7-point scale is quite meaningful. Thus, we find strong evidence that those who participated in the deliberative process significantly increased their knowledge on the arguments for and against the contested topic. However, it’s worth keeping in mind that there might be other factors during that time period that could’ve increased knowledge on these topics other than the deliberative process itself, so comparing the participants (treatment) to non-participants (control) is helpful to rule out those factors, which we discuss below.
ii. Participants reported more knowledge on the topic than non-participants
In addition to comparing changes among participants before and after deliberation, we also compared post-deliberation knowledge of deliberative process participants to non-participants. We were able to do this because all 105 badgeholders who voted in Retro Funding Round 4 answered these same survey questions as part of the post-voting survey (which took place shortly after the conclusion of the deliberative process).
If we simply compare the means of deliberative process participants (treatment group) and non-participants (control group), however, we’re not accounting for possible selection bias: Only half of the 50 randomly selected citizens ultimately chose to participate, and there are probably some systematic, non-random differences between those that accepted the invite and those that didn’t (for instance, maybe those who accepted the invite are more engaged, or have more time, etc). Those systematic differences might also influence the outcome of knowledge on the contested topic, as opposed to differences in knowledge being driven by participation in the deliberative process.
Effect of Deliberation on Knowledge (IV Estimation)
| Variable | Estimate | Std. Error | t statistic |
|---|---|---|---|
| (Intercept) | 5.113 | 0.148 | 34.54 |
| Deliberation participation | 0.381 | 0.414 | 0.92 |
| Adjusted R-Squared | 0.051 | ||
| Wald Test | 0.8455 on 1 and 103 DF |
To get around this, we ran an instrumental variable (IV) analysis, which essentially takes into account this selection bias and adjusts the estimates accordingly. With this model, we see there is about a 0.4 increase in knowledge (fairly substantive on a 7-point scale) but the relatively large standard errors suggest there’s some noise in the data (probably also due to the fact that n = 24 is a relatively small sample size). This noise, which results in a lack of statistical significance, might also mean that knowledge levels were not as distinctly different between both treatment and control groups because there were other factors in the broader environment that both groups were exposed to and that also influenced knowledge — such as spillover effects from the deliberative process “treatment,” e.g. public gov forum updates summarizing the arguments for or against deducting funding.
The table below shows the raw means of the control group (the 62 badgeholders who were not invited to participate in the deliberative process but who did vote in Retro Funding Round 4) compared to the “compliers” (the 24 badgeholders who participated in both deliberative process sessions) and the “non-compliers” (the 19 badgeholders who were randomly selected to participate in the deliberative process but did not choose to, but who did ultimately vote in Retro Funding Round 4). We see that there do seem to be some baseline differences in post-deliberation knowledge between the three groups.
Mean Knowledge by Compliance Group
| Subgroup | n | Mean Knowledge | Std. Dev. | Median Knowledge |
|---|---|---|---|---|
| Complier | 24 | 5.83 | 0.92 | 6 |
| Non-Complier | 19 | 4.67 | 1.24 | 4 |
| Control | 62 | 5.11 | 1.06 | 5 |
iii. Qualitative evidence suggests that the acquired knowledge came primarily from small group discussions, followed by the info kit
When asked “what arguments or information most influenced views on the topic during the deliberative process,” 15 of the 24 participants (62.5%) mentioned small group discussions in their free response entry, for instance:
- “A number of things specific individuals said in the small group breakouts I found most influential because they often filled in gaps in my knowledge”
- “Generally, the opportunity to hear other people’s perspectives during the breakout sessions”
- “Some of the discussion yesterday was helpful to broaden my view, but I didn’t really change my mind very much.”
This was followed by 6 out of 24 (25%) mentioning the information kits which were shared as preparatory pre-reading, for instance:
- “Information kit & small group breakouts were both incredibly helpful in advancing my thinking on the topic”
- "The information kit was very helpful for setting context. I wouldn’t say they influenced my views much though, but they helped clarify things for me. "
It’s worth noting that multiple responses explicitly said that while the deliberative process enhanced knowledge on the topic, it didn’t necessarily change their opinion. This aligns with the findings we discuss in a later section on whether deliberation seemed to increase consensus on the contested topic.
(2) Deliberation increases trust in opinions of other badgeholders
- Theory: Deliberative processes like citizens’ assemblies provide a space where participants can develop trust through dialogue and problem-solving, and past empirical work has found that deliberation can increase empathy and trust between participants as individuals are exposed to the lived experiences and perspectives of others.
Outcome to test Hypothesis 2: Deliberation Increases Social Cohesion
| Outcome: | To what extent do you trust the opinions of other badgeholders? |
| Scale: | 0 (very low trust) ... 4 ... 7 (very high trust) |
i. Participants had higher trust in the opinions of other badgeholders post-deliberation compared to pre-deliberation
| Paired T-test Results | |
|---|---|
| MeanPre-deliberation | 4.43 |
| MeanPost-deliberation | 5.13 |
| Mean Difference | +0.7*** |
| Standard Error | (0.17) |
| t-statistic | 4.04 |
| Degrees of Freedom | 23 |
Comparing pre/ post-survey responses, participants increased about 0.7 on a 7-point scale measuring their trust in other badgeholders. With a p-value of 0.001 this is a statistically significant shift in opinions among participants, and substantively moving nearly 1 point on a 7-point scale is quite meaningful. Thus, we find strong evidence that those who participated in the deliberative process significantly increased levels of trust in the opinions of other badgeholders.
ii. Participants had higher post-deliberation trust than non-participants
Effect of Deliberation on Trust in Other Badgeholders (IV Estimation)
| Variable | Estimate | Std. Error | t statistic |
|---|---|---|---|
| (Intercept) | 4.839 | 0.151 | 32.054 |
| Deliberation participation | 0.456 | 0.423 | 1.078 |
| Adjusted R-Squared | -0.009 | ||
| Wald Test | 1.162 on 1 and 103 DF |
Similar to above, we used IV estimation to account for potential selection bias given the opt-in nature of the treatment. We see there is about a 0.5 increase in trust in other badgeholders (fairly substantive on a 7-point scale) but again, the relatively large standard errors suggest there’s some noise in the data (probably also due to the fact that n = 24 is a relatively small sample size).
(3) Deliberation increases support for deliberative process itself
- Theory: Participants in deliberative processes will come to view deliberation as a legitimate and effective way to make decisions, meaning participants gain trust and appreciation for the deliberative process itself.
i. Among participants, 22 out of 24 (92%) were supportive of a deliberative process being useful for additional topics and even bigger settings
The table below summarizes responses to the question: “Do you think a deliberative process could be useful for additional contested topics and even bigger settings?” We see that the overwhelming majority of participants were supportive of deliberation as an effective governance tool.
Qualitative data source: This data comes from a free response question on the post-deliberation survey administered to participants.
ii. For both participants and non-participants alike, and for those for and against ratifying the definition of profit, there were consistent themes of positive support toward deliberation as an effective tool in the decision-making process
Qualitative data source: This data comes from governance forum discussion comments, and ratification voting data on Snapshot.
(4) Deliberation moved participants toward consensus on only one of four categories
- Theory: Structured discussions can lead to significant shifts in opinion and create more nuanced, consensus-based decisions because deliberation creates an environment which participants can engage in with diverse viewpoints and can help bridge ideological divides.
Outcome to test Hypothesis 4: Deliberation Increases Consensus
| Outcome: | Which of the following types of external funding do you think should be deducted from public goods funding rewards? |
| Select all that apply: |
|
The tables below show the aggregate pre/ post- survey results for the question about which types of external funding should be deducted for the 24 badgeholders who participated in both deliberative processes. For the categories of VC Funding, OP Grants, and Other Grants, there is no strong evidence of consensus post-deliberation. In the case of VC Funding, the majority vote switched in favor of being “against deducting” but remained at a close margin. For the category of Revenue, it does appear that opinions became more aligned, with only about 20% of badgeholders in favor of deducting post-deliberation, compared to 46% being in favor of deducting pre-deliberation. The pairwise t-test results for changes in opinion on deducting Revenue also confirm that this is a statistically significant shift. There is no statistically significant shift in the other three categories.
Support for Deducting External Funding (Pre-deliberation):
| VC Funding | OP Grants | Grants | Revenue | |
|---|---|---|---|---|
| In Favor of Deducting | 14 | 14 | 8 | 11 |
| Against Deducting | 10 | 10 | 16 | 13 |
| Total | 24 | 24 | 24 | 24 |
Support for Deducting External Funding (Post-deliberation):
| VC Funding | OP Grants | Grants | Revenue | |
|---|---|---|---|---|
| In Favor of Deducting | 11 | 15 | 6 | 5 |
| Against Deducting | 13 | 9 | 18 | 19 |
| Total | 24 | 24 | 24 | 24 |
What’s Unique about the Optimism Case?
There are some things unique to Optimism’s deliberative process that should be taken into account when interpreting these findings:
i. Highly engaged population
This deliberative process was conducted among Optimism’s badgeholders, a group of highly engaged governance stakeholders who skew technical in their professions and are based in approximately 20 different countries. Of the 50 badgeholders randomly selected to participate, the 25 who showed up were likely especially engaged as they made time to participate in two 90-minute deliberative sessions (and in most cases, attended a third optional 90-minute deliberative session) in addition to completing the pre-reading before each session. In some cases, participants joined despite being based in inconvenient time zones (e.g., joining from Asia where the local time was 2am). Throughout the process, participants repeatedly requested additional pre-reading homework and asked for a third deliberative session.
ii. Direct policy implications
Another unique feature of this deliberative process is the fact that, pending ratification from the full set of badgeholders, the proposed definition of profit decided upon during the deliberative process would be directly implemented into the upcoming Retro Funding Round 4. This is distinct from deliberative processes in other contexts that have been criticized for being performative, or at best advisory. Optimism, on the other hand, is a governance structure that progressively decentralizes over time, meaning badgeholders held immediate decision-making power in Retro Funding Round 4.
iii. Lack of clarity on the topic
Throughout the process, it became clear that the topic of whether external funding should be deducted from public goods funding rewards was a question that also contains value judgements, whereas the resulting policy was meant to focus on the practical implementation details of this question. For example, a discussion about whether VC funding should be deducted from public goods funding is different than a discussion about whether VC funding could be deducted given limitations in what funding data is available for projects and whether it’s possible to deduct accurately. There were also time constraints given that the new definition of profit needed to be ratified before the upcoming Retro Funding Round 4.
Recapping Key Takeaways
In sum, the key takeaways from this analysis are:
-
(1) Running another deliberative process could help to increase participants’ knowledge on the arguments for/ against the deliberated topic:
- There’s strong evidence that those who participated in the deliberative process became more knowledgeable about the topic, with badgeholders reporting a significant increase in their understanding of the arguments for and against the topic after engaging in the deliberative process.
- Comparing participants to non-participants, deliberative process participants tended to report higher knowledge than non-participants after the deliberative process, though whether deliberation alone is driving this difference is less certain (likely due to the small sample size of 24 participants, or perhaps spillover effects from the deliberative process also increased non-participant knowledge which makes the post-deliberation difference less meaningful)
-
(2) Running another deliberative process could help to increase participants’ trust in the opinions of fellow governance participants:
- There’s strong evidence that participating in the deliberative process led badgeholders to trust each others’ opinions more, with badgeholders reporting a significant increase in trust in the opinions of other badgeholders after engaging in the deliberative process.
- Comparing participants to non-participants, deliberative process participants tended to report higher trust than non-participants after the deliberative process, though again, whether deliberation alone is driving this difference is less certain (likely due to the small sample size of 24 participants)
-
(3) Running the deliberative process seemed to encourage support for the deliberative process itself as an effective decision-making tool
- This support for a deliberative process as a useful governance tool came from participants and the broader population of non-participants
- This support came from those who voted in favor and those who voted against the new definition of profit that was decided during the deliberative process
-
(4) Participating in the deliberative process only seemed to increase consensus on one of four categories under the deliberated topic
- This suggests that deliberation may foster agreement on specific aspects of a topic (perhaps through increased trust and information exchange, as documented above) though not universally across all categories, so this question probably merits further study
- This finding is likely also influenced by the practical constraints and lack of clarity surrounding the topic of the deliberative process, and the fact that the category (Revenue) which did see movement toward consensus is less polarizing and more clearly scoped than other categories (e.g., moral aversion to VCs, or lack of clarity on what is considered an OP Grant, etc.)
What This Means for Optimism Governance
We found evidence that deliberation can be a tool for knowledge exchange and increasing social cohesion, so this might make sense in contexts where comprehension of complex topics, or where bridging a divide between parties with low levels of trust, is important. Specifically, our findings suggest that a deliberative process might be useful for improving comprehension of complex documents like the Law of Chains or Blockspace Charters. A deliberative process could also be useful as a tool to facilitate conversation between the Token House and Citizens House in a way that improves trust in the other House and thereby reduces gridlock.
We found no strong evidence that deliberation shifted views toward consensus on the overall topic deliberated, suggesting that this type of deliberation tied to a binding policy recommendation may be less useful for the most polarizing topics, or for philosophical values questions which are difficult to coerce into a measurement question. With this in mind, we believe further exploration and study is warranted as there were unique considerations related to the specific topic that was discussed and which may have impacted outcomes. There are no immediate plans to implement another deliberative process experiment but we may explore this as appropriate.
Finally, we are very grateful to our facilitators who made this deliberative process happen: @Antoine Vergne (Missions Publiques), InĂŞs NW (InĂŞs NW), and Andrea Gallagher (RnDAO).
Thanks for reading and would love to hear any reactions in the comments below!
