Retro Funding 5: Expert voting experiment

[system]

Retroactive Public Goods Funding 5 (Retro Funding) will reward contributions to the OP Stack. A high degree of knowledge is required to understand the OP Stack’s different components, the impact of protocol development initiatives and the usefulness of tools that support OP chain operators. The hypothesis that round 5 is aiming to test is if experts make better OP allocation decisions on OP Stack contributions compared to non-experts.

Research Questions

The Retro Funding 5 governance experiment is designed around the following research questions:

1. Are there significant differences in how experts versus non-experts vote to allocate resources to technical contributions?
2. Are there significant differences between existing badgeholders and new guest voters, both in how they vote to allocate resources and in other characteristics?
3. Will sorting badgeholders into smaller groups dedicated to evaluating a specific set of applications improve the voter experience?

Expertise Measurement

Expertise is measured via a score assigned to each individual based on their past Github activity relating to the OP Stack. The GitHub accounts of guest voters are collected in the application flow to become a guest voter. Badgeholders are invited to add their GitHub account to their Optimist Profile in the process of completing Season 6 onboarding. The scores are assigned using an algorithm developed through a collaboration between Karma3Labs and Open Source Observer, and is based on Hubs and Authorities. The algorithm is more complex than EigenTrust, because it incorporates two entities: GitHub Repos and GitHub Users, both of which can have ‘trust’ and give ‘trust’ to each other. You can find more details on guest voter selection here and the algorithm here.

Voter groups

In order to account for a small degree of churn, the following voter groups have been formed and participants informed by email:

Badgeholders: 97 individuals

Guest Voters: 30 individuals

Guest Voters

• Guest voters were separately announced in this forum post.
• Guest voters were selected by choosing the top ranked applicants based on OP Stack expertise. The algorithm ranked all GitHub users on their proximity to the OP Stack.
  • The highest rank among the Guest Voters is #3 out of ~74,000 GitHub users with a score of 2.47 (zero is a perfect score)
  • The median rank was #111
  • The lowest rank among the Guest Voters is #861 with a score of 5.74
• 30 Guest voters were invited to account for the possibility of churn.

Citizens

Given that the number of Citizens that opted-in to participate in Season 6 was 97 (as of August 16th), all Citizens who opted-in before 16.08.24 have been invited to participate in the Round, with 34 assigned to the experiment condition (high OP Stack expertise) and 63 to the control condition (low OP Stack expertise).

• The expertise ranking of Citizens ranges from none (no GitHub or no relevant GitHub activity) to the highest ranking expertise score - #58 with a score of 4.17
• The median expertise score was 6.95
• Citizens above the median expertise score were assigned to the experiment condition and the other half was assigned to the control condition.

Random sampling

Random sampling is a widely used statistical method in which all members of a population (all Citizens) have the same probability of being selected. Random sampling does not guarantee that a particular sample is a perfect representation of the population, but rather allows for valid conclusions to be drawn about the entire population based on the sample. Another way of saying this is that the random sample approximates the full population. This is due to the equal probability of selection. Read more here Experimenting with Random Sampling in the Citizens' House

Treatment and control groups

Retroactive Public Goods Funding is an ongoing experiment, in which the Collective tests and validates different voting designs based on how well they perform at rewarding impact. Previous experiments were largely product iterations based on learnings from previous rounds and hypothesised improvements. For future rounds, experimentation will become more scientific to better understand cause and effect of different design choices. We believe this approach will allow the Collective to develop an industry-leading approach to metagovernance that will strengthen the design of the system over time. Retro Funding 5 introduces the concept of treatment and control groups. In experimental design, the “treatment group” participates in the experiment, while the “control group” does not (representing the status quo.) This allows hypotheses to be validated or invalidated by comparing the results of the treatment group to the control group. Specifically, taking this approach in Round 5 will allow the Collective to generate actionable insights about the performance of expert voters compared to non-expert voters.

Voting Process

An iteration on Retro Funding voting design is the allocation of voters into subgroups. Each subgroup only votes on a specific category of projects (e.g. Ethereum Core Contributions, OP Stack Research & Development, OP Stack Tooling). This has been a popular experimentation request and is one proposed methodology by which the Retro Funding voting process can be scaled to reward more projects. This change can be understood as a product iteration, as its validation largely relies on the change in performance data compared to previous rounds and badgeholder feedback, while there is no control group which would allow for more scientific comparison.

Process Overview:

1. Subgroups are established: voters are (randomly) assigned into subgroups, the subgroups are equal in size and remain representative of the different voter groups.
2. Category allocation: All voters vote on the allocation of OP among the three categories of the round scope. These votes will be public.
3. Impact Evaluation: Each voter votes on the amount of retro rewards all projects within their assigned category should receive. Votes will be private to prevent intimidation and coercion, promote unbiased decision-making and protect against bribery.

Voting design

The Foundation Mission “Evaluating Voting Design Tradeoffs for Retro Funding” is expected to deliver relevant insights into voting design tradeoffs, which will inform the selection of a voting mechanism for round 5. Further details on the implementation of impact = profit within the round, and the policy for conflicts of interests among badgeholders, will follow.

[joanbp]

This is exciting! Lots of interesting experimentation here.

Really cool that voters get to decide on the OP distribution between different categories after the application deadline when we know how many have applied and maybe have a general sense of the quality of applications. Maybe some useful data can be gained from the review process?

@system

Subgroups are established: voters are (randomly) assigned into subgroups, the subgroups are equal in size and remain representative of the different voter groups.

I’m curious - did you consider asking voters which category/categories they would be most keen to vote on? And if you did, what was the thought process behind choosing the random sampling approach here?

Also - I look forward to see what kind of interaction there will be between citizens and guest voters and between experts and non-experts! Aside from the question of whether some people are better equipped to vote because of their past experiences (which I would expect to be the case), I would hypothesize that introducing experts into the process could raise the level of all voters …but only if the conditions are conducive, and if experts and non-experts alike are willing to enter into fruitful exchange.

[Peersky]

Are there any means, at this time, to get (retro) grants support for a research or application work in the voting mechanisms themself?

[joanbp]

A couple of practical questions:

When will badgeholders and guest voters be notified of whether or not they have been sampled/chosen to participate in Round5?

Is there a timeline for reviewer signup?

And will there be any efforts to keep the remaining citizens in the loop - feeling like part of a meaningful group - apart from posts in this forum?

[Jonas]

@joanbp the main motivator for randomly assigning badgeholders to categories is to ensure that each category has the same mixture of voter groups and to reduce selection bias as well as concerns around conflict of interest.
The alternative of voters self selecting would require an additional step for voters to pick a category before the round starts and a selection process to decide which voters get assigned to their category of choice and which don’t, to maintain the mixture of voter groups.

Badgeholders and guest voters will be notified at the end of August if they are selected. There’s no timeline on reviewer sign up yet, addressing the feedback from the last process is still wip.

Citizens will be involved in shaping different components of the round, similar to last round. More details on this following soon

[Liliop.eth]

Hey @Jonas that´s a good experiment especially for non-technical profiles!.

I just have a few questions regarding to the timeline. You said in today’s meeting that the voter results will be released at the end of this month, does that mean the dates to apply have already passed and are those the ones you put in the other post? or is there an update of the dates? Can you clarify for me please?

How to become a guest voter for Retro Funding 5

Applications to become a guest voter are open until updated: new deadline is July 31st! to developers contributing to the Optimism ecosystem. Of all applicants, ~30 will be selected to participate as guest voters using a ‘Proof of Work’ selection mechanism (see below). Part of this mechanism will involve an analysis of public GitHub account activity before June 27th 2024. The algorithm will be shared publicly after the selection process has taken place to avoid influencing the experiment.

If you are a developer and you want to participate in voting for Retro Funding 5, please create your Optimist Profile and submit an application here by July 31th 2024. Any developer can apply, the algorithm will rank and select the top applicants based on their code contributions to relevant repositories. Therefore, please apply even if you are unsure whether your contributions are significant. Please do not apply if you are not a developer.

Thanks in advance!!
Lili

[optimistic_emily]

Hi @Liliop.eth the deadline for Guest Voter applications has passed, and the list of Guest Voters was announced today over in this post.

[Liliop.eth]

Hi @optimistic_emily, I see it, thanks for your reply!

[system]

This post was edited on 21.08.24 to include final numbers of participants in each experiment condition.

[norswap]

As a point of feedback, I think it’s very interesting I wasn’t selected as a guest voter. Having written most of the initial OP stack specs + having written a tool that helps deploy OP stack rollups, you’d have thought I was an ideal candidate.

I’m not complaining, this is just community service and can be time-consuming, so really no problem on my side.

I just wanted to point out that the algorithm used might have interesting flaws or bindspots. I didn’t really dig in the description, but just volunteering this data point.

EDIT: Just saw I was randomly selected — so it might be that I wasn’t under consideration for that reason. Worth checking in any case if the algo is intended to be used again!

[optimistic_emily]

Hey @norswap sorry for the lack of clarity!

You are a Citizen, so you were not considered by the algorithm that selected Guest Voters. You’re able to vote because you’re a Citizen (you should have received an email inviting you to vote). Guest Voters are non-Citizens who applied and were selected to vote in this Retro Round.

We did run the scoring algorithm on Citizens just to assign folks to experiment conditions (no effect on voting rights), and I just took a look at the results for you. Your score is very high - exactly what we would expect given the context you shared above.

[norswap]

Well that’s confusing because you sent me an email to apply to become a guest voter

All G though, thanks for the clarification!

[pfedprog]

Too many words and I just see the definition of the random sampling in the post. So on what data and how was it applied?

proposed methodology by which the Retro Funding voting process can be scaled to reward more projects

Ok, but where is the initial or null hypothesis that this actually gets done in the end.