Hey! thanks for the observation! I realized that the whole third graph was wrong gg, now is correct 
2 Likes
We participated in RetroPGF Round3 and I have some feedback to share.
What happened
Thereās one rule āDo Not DM badgeholdersā in this round. We are very confident in our project and followed the rule carefully to leave badgeholders alone. Our application has a minimum 500 DAU, it helps thousands of web3 users every day. But when we checked out the results, our application only got 4 ballots until 12/06.
Then we realized that this game is not played like that. So we tweeted about Retropgf and our ballots jumped to 12 in just one day. However, itās too late to do that and the deadline comes first than 17 ballots and we didnāt meet the quorum on time.
What we learned
From our experience, we learned that most badgeholders (90% of them) wonāt walk through all the projects one by one. They probably will spend more time on Twitter. This entire retropgf voting mechanism may not be efficient enough and the grants that 100% rely on badgeholders may not be a good idea. Who knows whether they have researched all the candidates?
Results
In such an unfair environment, many honest projects who trusted badgeholders and followed the rule failed to meet the quorum. However, many dishonest projects made it by contacting badgeholders and making promotions on Twitter. The āDo Not DMā rule is very misleading and I didnāt hear any project got disqualified by DM a badgeholder. I hope Optimism Foundation can review such situation and reconsider this voting mechanism to work in RetroPGF Round 3. Projects which got 10-12 ballots should be given some chances. I donāt think those ācheatersā can help Optimism Ecosystem grow further.
Thank you!
3 Likes
Great feedback, but, unfortunately thats how co-ordination games in web3 run usually. I have been active in web3 since 2021 through Gitcoin DAO.
And thats what I have noticed:
- Twitter engagement brings the most results.
- You can certainly contact and promote to voters after the round ends or before it starts.
- Some project interact with the voters directly on a daily basis and stoping the line of contact simply for fairness purpose is just not desirable.
2 Likes
I havenāt commented on RPGF #3 thus far because my thoughts have not changed from previous feedback and comments on the matter. But just to reiterate:
- Citizen House needs to vote in specific, objective qualifying criteria. Not important to get every project - more important to capture most valuable projects with minimizing overhead for badgeholders.
- Citizen House needs to be greatly expanded. As to how - thatās a matter for another topic.
2b) Token House should have a proposal type for vetoing specific RPGF recipients, as well as removing specific badgeholders. - Iād like to see all OP allocated towards RPGF be burned.
- RPGF will instead be awarded through a share of sequencer revenues.
7 Likes
This was our first time participating in RetroPGF. At CryptoConexion we are very grateful for the 32 badge holders who voted for us and the 4 who included us on their lists. Here are some thoughts on the process:
- As we were preparing our application, the content LauNaMu published on X helped us as we built our proposal.
- While no one on our team is a badge holder, we know how challenging it is to thoroughly evaluate over 600 diverse projects. Hats off to the badge holders.
- There seems to be a variety of opinions on the use of lists during the process. As a project, we did see the value of lists. I do want to call out the LatAm list created by Brichis- she provided individualized feedback to each project on improving impact in her impact evaluation. I donāt know if others did as the lists are no longer visible. Understanding the workload for badge holders, this level of personalized assessment is likely not feasible across the board, but the value is clear.
- We would suggest allowing lists to remain public even after voting concludes, maybe they are but not sure where they are located.
- The transparency from badge holders on X and Farcaster helped us understand the evaluation process. This document, which is a work in progress, captures some of the feedback we have found on X, we will continue to edit it this week: Notion ā The all-in-one workspace for your notes, tasks, wikis, and databases.
- Discovering information, resources and content across platforms is really hard both for badge holders and for projects, as we can see based on the content we collected from X. The Optimism team works so hard to create really valuable content - making it discoverable is key. Finding a way to connect everything would be very helpful.
- Yet on the topic of discoverability, we did get to know other public goods projects building in this space. We loved the UX/UI in Grow the Pie, as an example. I am sure they had record breaking traffic. But mostly, we discovered projects because there was so much shilling on X. I donāt think that was expected, or if it was helpful for badge holders, but it helped us connect with others.
We recognize that we will not know until January who will receive funding and how it will be allocated but we wanted to share our experience. Thank you again for the transparency and commitment to public goods. You could say we are Optimistic.
6 Likes
Hi 
@Michael ā¦ We love the the idea of paying a few OP or having the ability to allow someone else to stake OP in your favor for eligibility with a rather low threshold.
Here are a few other options we cooked up.
- The integration of KarmaHQ delegation dashboard & their new platform GAP aka the Grantee Accountability Protocol. Between governance participation being recorded onchain and track records from previous grant funding history a prerequisite could also be made for certain criteria to enter into RetroPGF.
Gitcoin has already implemented the use of this mechanism which includes making attestations for milestones related to grant funding amongst a variety of blossoming public goods funding ecosystems. Many of which users already cross pollinated over from the Optimism collective and back to RPGF3 such as Nouns grants prop house. 
Every grantee who receives RPGF of course is not obligated to do anything with the rewards from our understanding. Meaning they can spend the money on whatever they want.
It would be interesting to see how many projects from the most recent round 3 or previous rounds are willing to report on the GAP to ensure that others have a transparent way of seeing what the OP goes towards.
This is something that we have taken the initiative to do ourselves from the previous RetroPGF 2 funds our project received.
- Hypercerts are another type of impact certificate that we have used in the Gitcoin ecosystem to help report impact that has been made. From my understanding this will be utilized during the next round of Gitcoin to serve as a way to provide another tier of visibility for impact makers in the space beyond impact recorded with attestations via GAP by KarmaHQ.
- Curia Hub is something I certainly thought
would have been utilized a lot more in the process for governance participation in the RPGF review process.
Both KarmaHQ & Curia are funded by Optimism grants so that seems like 
a very big opportunity to integrate these systems vertically across the board. More layers of data 
& information that is readily available at the fingertips of the badge holders will simplify the decision making process when allocating retroactive funds. Seeing as how Optimismās RetroPGF program is aimed at the technical aspects of blockchain this just feels right to utilize a wide variety of tools to enhance the fairness of the process. Taking away the social media driven approach in the next round should definitely 
be a prioritization for RPGF.
1 Like
If projects use grant writing/promotion services then these should be disclosed, ideally with the amount/percentage being paid.
3 Likes
Another exciting round of RetroPGF 
and one step closer to a thriving ecosystem of public goods.
One common piece of feedback has been the natural divide between more popular brands and the hundreds of lesser-known projects. Typically how this plays out is that smaller projects miss essential funding, and well-funded projects (VC or otherwise) receive something more like a bonus.
A potential solution I see is splitting the RetroPGF pot into Popular & Underdog categories, helping to reserve a portion of funding for those that need it most.
To combat the sheer volume of projects, and help navigate any bias, badgeholders could be shown a smaller, randomized range of available projects, and have the proper time to add their analysis.
While I donāt imagine this has happened, randomized selection would have the secondary effect of making harassing/bribing individual badgeholders more difficult.
Thatās my food for thought! Lots of interesting ideas in here 
3 Likes
Hey guys, my feedback on this,
The threshold set for the ballots was too high and should be defined based on the total number of badgeholders who voted. If 100% voted, that would mean 17, and this value should decrease linearly based on the percentage of total votes. Lists disappeared on the platform on the day of the final voting; I donāt know if that could have influenced the result.
How do we define what could be considered a public good? There were several projects on the list with huge funding already, and they were the ones getting the most votes, obviously because they are more popular and have a greater impact. These projects should be in another league. With the current system in place, how can small projects become bigger if the only ones with already huge success will amass almost all of the funding?
We noticed late in the process that there was significant shilling on Twitter about showcasing the project. We later realized that the game is interacting with badgeholders. This should be highlighted, and projects should be better categorized based on what they are trying to achieve. Badgeholders should also be categorized based on their areas of expertise and should only be able to assess projects that are in their field.
2 Likes
I have shared my reflections here.
Iām still processing, and I decided to keep it in a separate thread to be able to add points if/when I think of them.
1 Like
Would suggest you cross-post this voting rationale!
2 Likes
Badgeholder Selection
The problem:
Any human coordination game is only as good as itās incentives. Thatās why we are building RetroPGF in the first place. But there is one area where there is no incentives alignmentā¦ the badgeholders.
What is the incentive for badgeholder to be the best possible voters in retroPGF?
One obvious way is that if the success of the experiment & the continued growth of The Optimism Collective a badgeholder personally. For example, any core developers of the OP Stack who are badgeholders obviously want to see the experiment succeed.
For those outside of the OP Ecosystem, its possible to still have alignment, but in this case they MUST understand the grander vision of RetroPGF. In other words, they need to deeply understand Etherās Phoenix.
My suggestions:
- A higher percentage of badgeholders we bring in should have a stake in the OP Collective. This means bringing them in from the various chains & protocols within the Superchain.
- All badgeholders should have some kind of orientation which involves testing their knowledge about Etherās Phoenix. This doesnāt have to be a super heavy thing, but itās important for badgeholders to at least be starting from a shared foundation.
15 Likes
Hello everyone, Iāve been reading some great feedback here and would like to share my perspective as well.
For context, this was my first round as a badgeholder, and it was also the first time a project Iām involved in participated in RPGF. Here are the main points I observed during this experience:
- Better definition of āimpact=profitā
I find this idea incredible, but thereās room for improvement in defining what āimpactā truly means. Initially, I focused on allocating OP to projects with an impact on the OP ecosystem, but as the round progressed, I noticed other badgeholders also rewarding impact on the entire Ethereum ecosystem.
While Iām not against this, I believe clearer guidelines for badgeholders could streamline the process. PS: The badgeholder manual was immensely helpful to me in this round!
Towards the end, I saw several projects that seemingly did little for the Collective surpassing the minimum quorum. Many still have a chance to earn more OP because a high score with 17 votes is more plausible than with +40 votes.
- Better criteria & communication
Communication improvement is essential. Many projects (and badgeholders) discovered the quorum of 17 during the allocation period.
Moreover, the review process in this round didnāt seem communicative enough. Many projects were flagged by the Foundation and/or the Collective without much information. On one hand, this is positive because many applications were processed, but on the other hand, itās not fair to the projects not to provide a coherent justification for the flag. Another project Iām part of was rejected, and I still donāt exactly understand the reason for it (Optimism Portugues).
I also believe allowing only 2-3 days for projects to appeal through a google form may not be the best option. Why not use the governance forum?
Moreover, I still believe that the quorum of 17 for a project to be āqualifiedā is not a good idea. In the last 2-3 days, several projects and individuals tried to contact me, essentially begging to add the project to my ballot so they could reach 17. I think this is not positive for the project, badgeholders, or the Collective.
- Set realistic expectations for badgeholders
The application said that āacting as a voter in RetroPGF 3 requires a minimum time commitment of 5-10 hours.ā I know it said āminimumā, but the time was much longer than that.
As mentioned by others, to do minimally acceptable work in this round, many hours of work were necessary, and 643 projects were at stake. There was no way to finish the work in 10 hours, especially if the badgeholder participated in pre-voting meetings.
It would be better to align these expectations before the Round starts for badgeholders to better plan their votes.
Also, the rule āDo not DM badgeholdersā should be taken more seriously. As long as we let it continue the rule will continue to be broken. Certainly, part of this arises from the need for smaller projects to promote themselves to reach the minimum quorum of 17. But canāt we create other forms of promotion/contact for these projects? Proof of Integrity conducted more than 7 initiatives for projects to present themselves; Why not do this on a larger scale?
Another important point is that it is almost impossible to analyze all projects. We need to be more focused and assign people who know about the subject to evaluate it. For example, itās not useful for someone who doesnāt know about development to evaluate development projects. Maybe we can think of ways to separate and specialize badgeholders more broadly, such as development, education, etc.
- Applications
I know others have already said this, but I would also like to see more info on applications. VCs are one of the things. Additionally, I would like that the unjustified absence of certain relevant information (such as VC investments) to be a criteria for disqualification of the project.
Not everyone will want to be transparent, but this needs to be penalized.
In the next round, perhaps there could be some form of reporting throughout the round, as well as a way for badgeholders to be aware of the report (much information was placed in the badgeholders group, but it easily gets lost there).
Also, as some others have suggested here in the forum, for the next round, it might be interesting to think about a token lock mechanism for the application of a project/individual.
- Rounds separated by category
It was quite challenging in this round, at least personally, to put projects with million-dollar investments on an equal level with projects without any funds. Individual applications were even more difficult.
The expected impact is totally different for an infrastructure project, and analyzing all at once certainly hinders the assessment of allocations.
Thatās it folks! Always remembering that RPGF is an experiment, everything I say is aimed at making things even better for the next round.
Thank you to all the fellow badgeholders who voted for their effort and dedication, and to everyone at the Foundation who conducted this round, it was an incredible experience.
13 Likes
Multiple language projects should apply as a Single project
8 Likes
As a first time badgeholder, I found it very meaningful to participate in such a wholesome and innovative governance structure. My favourite part was learning about all the projects that have been impacting the ecosystem! I see so much potential, but at the same time I found it quite overwhelming at times. Some specific feedback below:
This round was too big. It needs more facilitation or needs to be broken down
~640 projects, 30m OP, ~150 badgeholders. With the only formal coordination being Lists, it was extremely overwhelming. I fear that only a few badgeholders did deep dives and created lists and the majority of badgeholders relied on lists. By doing this, we are not harnessing the full potential and diversity of our badgeholders pool.
There are many potential solutions for this:
- Break rpgf into smaller rounds with a more clear focus. I think smaller projects would benefit a lot from this.
- Intentionally divide and conquer. Divide the projects into subcategories and then divide the badgeholders into small working groups (by expertise and/or interest). Each working group can tackle 1 or 2 subcategories. Ensure that each category is covered by 2+ working groups to ensure we get multiple opinions. This ensures ALL projects get fairly looked at. Each working group can then post their lists and rationale. This will encourage intentional collaboration and discussion amongst badgeholders. It will give us all a thorough, comprehensive, and diverse collection of opinions to base our individual votes on. It will also make the process less overwhelming because you can focus on a smaller subset of projects, knowing that collectively we will do justice to all the projects.
Voting Mechanics
Not everyone allocated 30m OP. However, this means that those who allocate the full 30m have more of an influence on the outcome. If the goal is to allocate 30m OP, than peopleās votes should be scaled to 30m if they allocated less (or there should be an option to do that on the voting app). However, if the goal is to allocate whatever feels appropriate (up to 30m) given the submissions, then I think using the votes as is (regardless of the total allocated is sensible).
Same goes for the quorum of 17. I think it would be better to decide what the quorum should be after the fact so that it doesnāt impact peopleās votes too much. Especially since some badgeholders didnāt vote.
VC Funded Projects
This was quite a conflicting topic this round. That being said, VC funded projects do deserve recognition and perhaps even some rpgf funding. However, grouping these with the other projects made it hard to figure out how to distribute funds fairly between teams with funding and teams without. If the goal is to reward impact retroactively (and create a positive feedback loop that encourages teams to strive for more impact), I think itās important to make sure the lesser known teams without VC funding get prioritized. This would be much simpler if we had two different rounds (one for VC-funded organizations and one for bootstrapped orgs).
Communication on Telegram
It was overwhelming to follow the communication in Telegram. A simple solution would be different topics for different types of comms. An announcements channel for critical information from the organizers, a channel for lists, a channel for philosophical discussions on public goods, etc.
** Badgeholder Expectations**
As discussed by others here, I think the expectation of 5-10 hours was far too low. It took significantly more time to do justice to the projects (and even then I felt like I should have done more). A more facilitated process could reduce the work everyone does while ensuring we have a more effective outcome.
7 Likes
Hereās what I would love to see in RPGF4, to make the voting experience better. Ordered in order of priority:
- We should have the option to write notes on applications for ourselves (i.e. non public). And e.g. have buttons to indicate if we want to vote for them or not (āYesā, āNoā, āMaybeā or blank), and then use that to filter applications. E.g. some projects I could see right away I didnt want to vote for, and I wanted to be able to mark them as āNoā to hide them going forward. This would allow me to rely less on working on my own Excel sheet.
- Import allocations via csv
- I would love to see something like Twitterās āCommunity notesā, or more specifically, badgeholder notes, on application pages themselves. E.g. there were a number of projects that did not list all their funding, I would like to be able to publicly add a note to that section and include sources showing that they raised more money. Sure this stuff can be brought up in forums or Telegram instead, but it gets lost. Put it right on the application itself.
Shout out to Jonas and LauNaMu for being amazing stewards of this process, loved how hands on you were with badgeholders!
5 Likes
Just posted a long write-up on the psychology of the game. It references various comments from this thread and proposes some ideas for how we can get better at playing this game.
Hereās the last paragraph, which sums up my perspective as a badgeholder:
The voting experience should be optimized to support voters in testing and implementing a well-defined strategy. At the end of each round, voters should have a clear understanding of the process and feel confident about their participation and choices. Speaking from personal experience, I am uncertain about my own effectiveness in the game. Despite investing time in identifying worthy projects and avoiding less desirable ones, Iām unsure if I maximized the potential of my top picks. Thereās a need for a system that helps voters articulate and evaluate their strategies clearly, as this is key to getting better at a repeated game.
(As a data and a mechanism nerd, I find the process of building a better game endlessly fascinating ā¦ but as a badgeholder and project participant, I donāt want to spend my life thinking about RetroPGF.)
14 Likes
Badgeholders who ālikeā a project rather than āloveā an impactful project are likely to bring down a projectās median.
In retrospect, it would have been more effective to focus on removing bad apples at the start rather than searching for good apples at the end.
Yes, definitely makes sense.
But, unfortunately, in case of penalizing vc funded such as those raised by @lefterisjp, the strategy would be to include them in a ballot with the least OP amount possible.
1 Like
First and foremost, I want to express my appreciation for the awesome UI experience provided by RetroHub, GrowthePie, Agora, West, OSO, RetroList, and others in RPGF round 3. Your dedication to simplifying processes and delivering valuable insights greatly eased the workload for Badgeholders.
I would also like to commend the pre-reviewers for their efforts in narrowing down the applications from 1673 to 643. This was a crucial step in managing the high volume of submissions, and their diligence cannot be overstated.
While there is room for improvement, I consider #3 of RPGF to be a success. This was my biggest contribution to the Optimism Collective to date, and I was determined to give all applications an equal and fair opportunity.
One concern I have is the self-nomination process. The prevalence of requests that are motivated more by financial gain than genuine contribution suggests a need for a more robust vetting system. To address this, one possibility, among many, would be to refine our nomination process. Implementing a two-stage voting system, where we first curate a list of RPG candidates and then allocate funds, could promote more informed and fair decision-making.
In my analysis of the applications, I aimed to maintain objectivity and fairness, with a focus on the realistic allocation of funds. However, it has become apparent that some applicants have inflated expectations regarding Impact=Profit. Lists included substantial sums for tasks that traditionally might not command such high rewards. This trend is concerning, as it may indicate a shift in focus from community impact to personal gain. Such a discrepancy between effort and expected reward could jeopardize the sustainability of our collective efforts.
I appreciate the opportunity to participate as a Badgeholder in round 3, and if given the honor to participate in round 4, I will bring the same realistic approach combined with everything I have learned in round 3.
Once again, Thank you to all the builders who contributed useful tools this round!
4 Likes
A way to filter the projects would be ideal, also i had issues when i was voting on projects that happened after i voted and alloted what i thought there allotmwnt of the 30m op tokens inwas allowed to designate to projects was after i voted/added them to the ballot, the programming wanted to bring me back to the beginning of the list everytime. Maybe i was doing something wrong but im unsure as i noticed that happened everytime and i would have to scroll all the way back down to the projects i had already voted on to surpass them and view/look at the projects i had not. It was rather teadious and bothersome to do so. It would definitely be awesome if there was a way to structure the architecture to where we wouldnt have to do so
2 Likes