# Simulating RetroPGF3 Voting and Funding Allocations

**URL:** <https://gov.optimism.io/t/simulating-retropgf3-voting-and-funding-allocations/7274>\
**Category:** Retro  Funding Missions\
**Tags:** round-3\
**Created:** [December 21, 2023, 4:28pm UTC](https://gov.optimism.io/t/simulating-retropgf3-voting-and-funding-allocations/7274 "2023-12-21T16:28:13Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![amy](https://avatars.discourse-cdn.com/v4/letter/a/4da419/32.png) [@amy](https://gov.optimism.io/u/amy)\
**Post date:** [December 21, 2023, 4:28pm UTC](https://gov.optimism.io/t/simulating-retropgf3-voting-and-funding-allocations/7274/1 "2023-12-21T16:28:13Z")

</div>

After reviewing all the RetroPGF 3 discussions, @ccerv1 and I collaborated on a simulator to analyze a few questions around the quorum-median model and enable badgeholders to further explore their questions and ideas.

This is a two-part post. In this first part, we describe the [“quorum-median” model](https://gov.optimism.io/t/simulating-retropgf3-voting-and-funding-allocations/7274#round-3-model-1) + general goals and assumptions, introduce a [simulator](https://gov.optimism.io/t/simulating-retropgf3-voting-and-funding-allocations/7274#simulation-4), and using the simulator with placeholder-data, attempt to [answer a few questions](https://gov.optimism.io/t/simulating-retropgf3-voting-and-funding-allocations/7274#quorum-median-model-questions-8) raised by the OP Collective. The second post, coming in January, will explore collusion and lists when the complete dataset is released.

Let’s start with laying out the “quorum-median” Round 3 model:

## **Round 3 Model:**

- Simple quorum of 17 votes + median of amounts allocated per project
- Lists created by badgeholders containing projects with suggested allocation distribution.

More details: [Round 3 Design post](https://gov.optimism.io/t/retropgf-3-round-design/6802) and [Badgeholder’s Manual](https://www.optimism.io/badgeholder-manual), and [V1 of results calculation](https://github.com/ethereum-optimism/op-analytics/tree/main/rpgf_calculator).

## An Ideal Model Would Solve For:

- Preventing badgeholder collusion (creating a threshold of requiring more voters to impact results).
- Projects with fewer ballots are not penalized and reduces a popularity contest. Smaller/independent projects should have equal chance of being reviewed/rewarded for their impact.
- Final allocation is closest to fill the gap of “impact = profit”, which means not more or less.
- Reducing the “metagame” that badgeholders engage in (making adjustments based on what they think others will do).

## **Assumptions:**

- Badgeholders _ **should** _ try to allocate the amount that fills the gap “impact = profit”
- Every badgeholder is _ **not** _ reviewing/voting for every project.
- Each badgeholder has 30M OP to allocate, but _ **does not** _ have to deploy the full amount.
- A 0 ballot _ **should not** _ be the same as an abstain/happy to go with majority’s decision ballot. An “abstain” ballot confirms review / approves project for quorum, but doesn’t indicate a specific allocation amount to be counted as part of the median.
- A project must meet a minimum of 1,500 OP to be funded. Therefore an abstain ballot means the project _should_ be funded, but more than 1,500 OP.
- A project can reach quorum _ **and** _ have 0 OP allocated.

There were a few discussions around the downsides of a quorum+median model, such as a “meta-game” (certain intentional or unintentional choices bringing down a project’s median) and [the irony of quorum collusion](https://twitter.com/ricmoo/status/1733148821578740044?s=46&t=4AO0rTqq96dYfzxweu6_8Q). This left us with a few questions we wanted to answer through simulations.

* * *

# Simulation

Carl created a [model](https://github.com/opensource-observer/insights/tree/main/experiments/retropgf_simulation) that allows you to simulate different round assumptions. It’s not very sophisticated, but should allow some experimentation around scoring algorithms.

## Initializing the model

First, initiate the round by setting the `max funding` equal to 30M tokens.

```python
simulation.initialize_round(
    max_funding=30_000_000
)

```

Then, generate a random set of projects.

```python
simulation.randomize_projects(
		num_projects=643
)

```

When projects are generated, they are assigned an `impact rating`. Impact ratings are distributed randomly along a simple bell curve.

Finally, generate a random set of voters.

```python
simulation.randomize_voters(
    num_voters=150, 
    willingness_to_spend=1.0, 
    laziness_factor=0.5, 
    expertise_factor=0.5
)

```

This function initializes a set of 150 voters with the following properties determined at random:

- The voter’s `willingness to spend` up to the maximum amount. A value of 0.5 would mean that some voters will only spend up to 15M tokens.
- The voter’s `laziness` when it comes to reviewing all of the projects. The higher the value, the fewer projects the average voter will include in their ballot. A value of 0.5 means that voters will select up to 50% of the projects in the pool.
- The voter’s `expertise` when it comes to identifying high impact projects. The higher the value, the more likely it is that the voter is making good choices and not just selecting randomly. A value of 0.5 means that voters will be in a calibration range of 50-150% of a project’s true `impact rating`.

The higher these three values are, the more likely it is that voters will deliver a very skewed allocation result, rewarding only a small number of high impact projects.

The model has some scaffolding to map relationships between voters and projects and model collusion among voters, but this hasn’t been implemented yet.

## Running a simulation

When a simulation is run, each voter effectively “scores” each project. The score is the product of their `expertise` (randomized within a range) and the project’s true `impact rating`. The scores are sorted and the bottom ones are thrown out based on the voter’s `laziness`. The voter iterates through each project and casts a vote. The vote itself is also a random number, between the `min funding` amount and a max variable that is set based on `max funding` and the total number of projects the voter is willing to consider. The max the voter is willing to give to any project decays after each vote. This continues until the voter runs out of money.

Once all votes are complete, the simulation calculates the appropriate allocation to each project based on different scoring formulas, eg:

```python
simulation.simulate_voting_and_scoring(
    n=1,
    scoring_method='median',
    quorum=17,
    min_amount=1500,
    normalize=True
)

```

The variable `n` represents how many times to run the Monte Carlo simulation in order to get a reasonable sense of the distribution patterns. The other variables include several variants of the `scoring method` (median, mean, simple quadratic funding, and [Ricmoo’s “outlier” method](https://twitter.com/ricmoo/status/1733148821578740044?s=46&t=4AO0rTqq96dYfzxweu6_8Q)), a `quorum` threshold, a `min_amount` per project threshold, and whether to `normalize` the results pro rata to allocate the entire pool of funding (ie, 30M OP). These are effectively all of the scoring inputs that voters were presented to voters and projects prior to the start of the round.

# Base case

We’ll use the settings above as our base case.

Here are the outputs of the model.

```auto
scoring_method median
vote_quorum 17
min_amount 1500
normalize True

num_projects_above_quorum 451
avg_payout 66507
median_payout 35367
max_payout 439910

```

This is what the overall distribution curve looks like.

 ![Untitled](https://europe1.discourse-cdn.com/bc41dd/original/2X/c/cd7fdffc5640804f227d775f0555e67bb3ba2ecf.png)

And we can see decent correlation between “impact” and tokens awarded (ie, “profit”).

 ![Untitled2](https://europe1.discourse-cdn.com/bc41dd/original/2X/6/60b3880f690f16b323f2446bda01c8ff0bdfd0a7.png)

* * *

## “Quorum-Median” Model Questions

Using placeholder data, we used this simulator to answer a few questions raised by badgeholders. We explored the impact around these factors:

- Total OP distribution amount
- Number of projects reviewed
- Quorum minimum
- 0 ballots
- Different models (ie. mean, quadratic formula)

The questions below are based on **the quorum-median model as default.** Our analysis:

1. Each badgeholder had $30M OP to distribute. We did not have to deploy all of the funds. **What is the impact if badgeholders deploy all of the $30M vs less?**

2. There were 645 projects to review, but badgeholders were not expected to review all of them. There was no set consistent nor minimum or maximum of projects per badgeholder to review. **What is the impact if badgeholders review lots of projects?**

3. A project must meet [a minimum of 1,500 OP to be funded](https://plaid-cement-e44.notion.site/How-to-Vote-6f4d199c20184e8d9ea5a1fc8657af8e). **What is the impact of a 1,500 OP minimum?**

4. Round 3 utilizes a median calculation. Keeping the quorum requirement the same, **what is the impact of running the _mean_ on the same dataset? What about Quadratic funding?**

 ![Untitled3](https://europe1.discourse-cdn.com/bc41dd/original/2X/f/fd975afd3b07919264a6b3e832d3b9062b489f94.png)

The quadratic funding model produced a very different distribution, with a much higher median but the top project receiving much less as well. This could be a good scoring method for rewarding a plurality of high impact projects and avoiding an overly long tail.

````
```
**scoring_method quadratic**
vote_quorum 17
min_amount 1500
normalize True

num_projects_above_quorum 454
avg_payout 66012 
median_payout 42259 (++)
max_payout 338246 (--)
```

````

 ![Untitled4](https://europe1.discourse-cdn.com/bc41dd/original/2X/3/3c711ed019d7003fe4ee8e4ac7d1bfbda2a8e54f.png)

1. A project needed a quorum of 17 votes. **What is the impact of a higher quorum of 34 votes?**

2. Badgeholders had the choice to “abstain” by not voting (which [expresses indifference as to how much OP a project should receive](https://www.notion.so/6f4d199c20184e8d9ea5a1fc8657af8e?pvs=21)) or vote with a ballot of 0 OP distribution (”0 ballot”). **Assuming the project meets quorum, what is the impact of a 0 ballot on a project?**

## Conclusion

These are starting points for the OP Collective to gather insights. We hope you will use this simulation to play with your own models and questions. We’d love to hear about your results and if you have any comments or questions, please leave a comment below!

---

<div class="post-metadata">

**Author:** ![chom](https://dub1.discourse-cdn.com/bc41dd/user_avatar/gov.optimism.io/chom/32/7294_2.png) [@chom](https://gov.optimism.io/u/chom)\
**Post date:** [December 22, 2023, 4:23pm UTC](https://gov.optimism.io/t/simulating-retropgf3-voting-and-funding-allocations/7274/2 "2023-12-22T16:23:12Z")

</div>

Would be great if could simulate a scenario where we take the sqrt on each project’s score before normalization.

Such as

- Project A got 900 votes → sqrt 900 = 30
- Project B got 100 votes → sqrt 100 = 10
- Project C got 25 votes → sqrt 25 = 5

And then normalize them to 30M OP

---

<div class="post-metadata">

**Author:** ![spengrah](https://dub1.discourse-cdn.com/bc41dd/user_avatar/gov.optimism.io/spengrah/32/11946_2.png) [@spengrah](https://gov.optimism.io/u/spengrah)\
**Post date:** [December 23, 2023, 4:44pm UTC](https://gov.optimism.io/t/simulating-retropgf3-voting-and-funding-allocations/7274/3 "2023-12-23T16:44:51Z")

</div>

This is fantastic! Thank you @amy and @ccerv1 for asking these questions and running this simulation. I think a robust simulation framework will be super impactful for designing and evaluating future rPGF experiments.

With that in mind, I’d like to make one suggestion. I think its worth playing with the assumption that impact is normally distributed. I suspect this might influence some of the findings in your analysis. For example, my prior is that impact follows more of a power law / pareto distribution, and that would likely alter your median / mean analysis.

---

<div class="post-metadata">

**Author:** ![NathanVDH](https://dub1.discourse-cdn.com/bc41dd/user_avatar/gov.optimism.io/nathanvdh/32/7825_2.png) [@NathanVDH](https://gov.optimism.io/u/NathanVDH)\
**Post date:** [December 26, 2023, 2:10pm UTC](https://gov.optimism.io/t/simulating-retropgf3-voting-and-funding-allocations/7274/4 "2023-12-26T14:10:30Z")

</div>

I’m really glad someone is looking into the allocation system of OP tokens in RPGF. I think that the median choice was wise and mean/quadratic doesn’t recognize that a few projects are really creating most of the really important tools/content in the ecosystem.

However, I think this also brings to attention the problem in my mind with badgeholders both choosing which projects are worthy of RPGF **and** the amount of OP tokens they’ll receive. Take a trial with a grand jury for example. Traditionally, most significant trials are decided by a jury making a yes/no decision, followed by a judge determining the sentencing duration. Here, the jury also picks the amount received but it doesn’t seem like it has the necessary tools to be able to make the right decision.

I think something along the lines of participatory budgeting experiments might prove better at allocating OP tokens to projects chosen by the badgeholders. As long as the process is really transparent and the criterias for judging impact are well thought out obviously.