In 2024 we ran two experiments to help create insights that could be used in developing an evidence-based Citizen selection mechanism in the future. Taking an experimental approach, we formed hypotheses about possible selection mechanisms and tested them by selecting Guest Voters using these selection mechanisms.
A key responsibility of Citizens is to allocate resources based on what will bring the greatest long-term benefit to the Collective. In the two Guest Voter selection experiments conducted in 2024, we focused on testing different selection methods with a primary goal of accurately rewarding impact. Both Guest Voters and Citizens participated in the token allocation task, and their performance was evaluated by having stakeholders vote on which allocation they believed was most accurate. We hypothesized that engaged, competent and aligned voters would produce the most accurate Retro Funding allocations, and attempted to measure these characteristics as intermediate outcome measures in the selected participants. We also hypothesized that insular groups would exhibit more vote clustering than groups with more diverse social graphs.
Guest Voter participation in Retro Funding 5
Retro Funding 5 rewarded contributions to the OP Stack and its dependencies with 8M OP in October 2024. 28 Guest Voters were invited to participate to help answer the research question:
Are there significant differences in how experts versus non-experts vote to allocate resources to technical contributions?
Guest Voters were selected based on their ranking on an algorithm designed to measure their proximity to the OP Stack. We refer to this selection method as ‘proof of work’, because the algorithm ranked participants based on their contributions to relevant repositories. We hypothesized that those selected via proof of work would demonstrate relevant competencies, engagement and alignment with Optimism. The Guest Voters were considered ‘experts’ for the sake of the experiment. Citizens were also asked to link their GitHub account to their profile so their score on the algorithm could be used to categorise them as ‘experts’ and ‘non-experts’. In addition, both Guests and Citizens were asked to complete a survey to collect further data.
In-depth analysis of the voting behaviour of ‘experts’ and ‘non-experts’ indicated that there was indeed a difference in how they allocated resources to technical contributions. A full analysis carried out by Open Source Observer can be found here. In addition to this analysis, stakeholders were asked whether the allocation made by Citizens or Guests was more accurate without knowing which was which. 12 Citizens, 13 Guests and 3 Superchain partners responded, with 70% preferring the allocation made by experts.
These findings are not surprising - those with a greater familiarity with OP Stack-related projects found it easier to accurately assess their impact. Voting patterns indicate that non-experts were more likely to make flatter allocations, suggesting that they didn’t have a strong opinion on the difference in impact between projects.
In addition to the main findings, several ad hoc insights emerged from Round 5. Guest Voters were highly engaged with the process and expressed a strong interest in participating again. Both Citizens and Guest Voters rated their experience with Guest Voter participation very positively, with a median score of 5.5/7 for Citizens and 6/7 for Guests. These high ratings reflect the overall satisfaction and enthusiasm for continued involvement in the process.
Round 5 Key Takeaways
- Guest Voters had stronger opinions regarding OP Stack contributions, as indicated by more varied distributions rather than flat ones.
- The allocation made by experts was viewed as more accurate by a majority of surveyed stakeholders.
- Both Guests and Citizens were highly satisfied with the participation of Guests.
Guest Voter participation in Round 6
Retro Funding 6 rewarded contributions to Optimism Governance with 2.4M OP in November 2024. 76 Guest Voters were invited to participate based on a process of random selection with opt-in. Community members were able to volunteer themselves on socials and a subset was randomly chosen.
By selecting a random group of community members to participate, we wanted to understand whether the lower level of connectedness among the Guest Voter group led to any differences in vote clustering. The participation of a random set of community members also allowed us to collect survey data on this group and use it as a control against which to compare Citizens and Guests from Round 5.
An analysis of vote clustering in Round 6 revealed that Citizens’ votes were more closely grouped than Guests’ votes, but only in the Governance Analytics category. This is likely due to Citizens sharing similar opinions on the impact of Governance Analytics tools, as they have experience using them.
Choosing Guest Voters from the community in the Governance round also meant that Citizens could be perceived as ‘experts’ in this round, with Guest Voters as ‘non-experts’. A blind poll of which allocation was more accurate indicated that Citizens and Optimism Foundation employees thought the allocation made by Citizens was more accurate, whereas top 100 delegates thought the allocation made by Guest Voters was better. 18 Citizens and 16 Top 100 delegates participated in the poll.
Additional ad hoc insights from Round 6 highlighted a few important observations. Initially, participants were randomly selected from all followers of the Optimism Collective on social media. However, this approach to random selection proved nearly impossible, as only 2% of those selected opted in. In contrast, when participation was made open to the community, allowing volunteers to express interest before being randomly chosen, the response was overwhelming, with over 200 volunteers signing up within just two days.
Characteristics of Citizens and Guests in Rounds 5 and 6
Additional data collected about Citizens and Guest Voters attempted to assess whether these groups differed on any of the following measures: alignment, engagement, insularity, competence. These indicators were hypothesized to be predictive of performance on the Retro Funding allocation task.
Results indicated that Round 5 Guest Voters and Citizens had a similar level of engagement in the Collective as measured by having received token awards for Mission Requests, Retro Funding and community contributions. Round 6 Guest Voters were slightly less engaged with Optimism, but still 26% worked for organizations that had previously received Retro Funding (more than 50% of Round 5 Guests and Citizens have received token awards for community contributions or Retro Funding). Guests in both rounds were also similar to Citizens on willingness to contribute time to the Collective. Overall, the data suggests that the Guest Voters in both rounds and Citizens are similar and likely come from the same population. There were two main differences that stood out:
- Round 5 Guests had a significantly higher OP Stack score and expressed more of an interest in the OP Stack and the applications built on the Superchain, whereas the Guests in Round 6 and Citizens were more interested in the Optimistic Vision and had lower OP Stack scores
- Guest Voters represented a less cohesive social graph than Citizens, with Citizens having the highest average trust score among each other.
Key Takeaways
One key insight from Rounds 5 and 6 is that expertise and familiarity with the domain led to more accurate Retro Funding allocations. However, this should not be taken to mean that only individuals with high domain knowledge should be responsible for making decisions about Retro Funding.
Instead, we can break down the Retro Funding allocation process into multiple smaller decisions, where expert input is more valuable in some areas than in others:
- Setting objectives to move the North Star
- Dividing the budget between objectives
- Defining the evaluation algorithm for a specific category of work
- Measuring the impact of specific projects
Experts likely performed better at the allocation task due to their deeper knowledge and familiarity, which made it easier for them to define and measure the impact of specific projects. The updated governance framework enables us to break the Retro Funding allocation into distinct decisions, each of which could be handled by a tailored decision-making process with different groups of voters. In the previous Retro Funding framework this type of tailoring was not possible.
A second key insight is that the groups participating in Rounds 5 and 6 closely resembled Citizens in many ways, often reflected in the Enthusiast and Bootstrapper personas. In Round 6, we also discovered that attempts to create a more random group of participants were largely unsuccessful. Ultimately, opt-in bias led to a voter group that was not significantly different from Citizens. While opt-in bias is not inherently negative—since a strong desire to participate can reflect alignment and engagement with the Optimism Collective—it also means that the resulting group may not fully represent the broader range of opinions that could be considered in decision-making.
A third insight that emerged throughout Retro Funding in 2024 was the conflicting feedback from participants. Some felt that the allocation process was too time-consuming, potentially deterring future participation, while others expressed a desire to spend more time on the task. This aligns with a key finding from the persona research, where some personas were eager to contribute more deeply to governance.
If the role of being a Citizen required a significant time commitment, it could limit who is able to participate and result in an overrepresentation of some types of stakeholders. However, there are many governance tasks within the Collective that could support deeper contributions. By separating Retro Funding into individual decision modules, we can expand participation in decisions that require broader input but less time, while allowing for deeper involvement in tasks that demand more specific knowledge, expertise, and a greater time commitment.
Next Steps
The insights gathered from the Guest Voter selection experiments in 2024, alongside the personas research and updates to the Retro Funding process, offer a solid foundation for the next phase of experimentation with voter selection and improvement to governance processes. As we move into 2025, we plan to build on these learnings to run further experiments with voter selection for Retro Funding, with the ultimate goal of developing a process for Citizenship expansion.
Additionally, we will continue to test and optimize the new modular structure of Retro Funding. This approach will help us better align participation with the specific demands of each decision, making the governance process more inclusive and effective.
Stay tuned for further updates as we implement these next steps and continue experimenting with innovative ways to enhance participation and decision-making in Optimism governance.