RetroPGF 3: Application Feedback

[system]

This thread is to collect feedback on the RetroPGF 3 application process

Applications for Retroactive Public Goods Funding Round 3 are live! You can apply here and find guidelines on the application process here. Applications will close on Oct 23rd.

Iterating on Round 2 learnings

Based on the learnings and reflections from RetroPGF 2, we identified the challenge of gathering more high-quality data on projects to evaluate impact & profit. Feedback from Round 2 highlighted the need for greater alignment between the application form and the voting assessment process.

See RetroPGF 3: Round Design for an overview of the iterations that have been made based on Round 2 learnings.

RetroPGF 3 Applications - gathering the correct data from Projects

The Optimism Collective is built on the axiom of impact=profit — the principle that positive impact on the Collective should be rewarded with profit to the individual.

To achieve this vision, the RetroPGF process should measure the impact projects provided to the Collective and the profit projects already generated from their impact. RetroPGF rewards are meant to fill the gap between a projects impact to the Collective and the profit they already received for that impact.

The RetroPGF application form enables projects to self-report their impact & profit and provide references to relevant data sources. While the Collective has defined broad impact categories for RetroPGF, projects often have a more detailed understanding of how to measure the impact of their work, and it is important to empower them to surface their impact to badgeholders as they see fit.

Impact statement

The RetroPGF application features a section called “impact statement”, in which projects can describe their contributions and the impact they provided to the Optimism Collective.

See screenshot of the impact statement

The impact statement includes the following fields:

• Describe your contribution(s) to Optimism: A free-form text field in which projects can describe their contributions
• Contribution links: Projects can highlight external resources relating to their contributions
• Impact category: Projects are asked to pick one or multiple of the Round 3 categories that best describe their impact
• Impact description: A free-form text field where projects can describe the impact their contributions provided to the Optimism Collective
• Impact metrics: Projects can highlight qualitative or quantitive metrics that highlight the impact of their contributions

Allowing projects to reference standardized external resources, such as Github Repos and onchain contracts, can power community-built data analytics tools to calculate projects’ impact using a variety of different approaches and formulas.

Impact metrics try to support the evaluation process by giving badgeholders measurable insights into the impact of a project, instead of solely relying on qualitative descriptions.

Grants & Funding

The Grants and Funding section tries to collect information relevant to the profit side of the impact = profit equation. The main goal is to understand the funding projects have already received for their impact.

See screenshot of the Grants & funding section

• Projects can add one or multiple grants and funding sources.
• For each grant/funding source, they can select from different categories (Governance fund, Partner fund, RetroPGF 2, RetroPGF 1, or “other”) and provide an amount and a description of the grant/funding they received.
• While projects can volunteer information on investments they received, in this round the application form and guidelines do not require projects to report this information. The reasoning behind this is that venture investment is made in exchange for ownership in a project and, therefore, should not be considered as profit.

The application form does not collect information relevant to means testing (see Vitalik’s Review of Optimism retro funding round 1). Means testing tries to understand the means a project has at their disposal and how reliant projects are on RetroPGF rewards to continue their work. By rewarding with the expectation of future work, means testing leans more towards proactive grant giving. As the main experiment here is to reward contributions retroactively, RetroPGF 3 experiments with deprioritising collecting information relevant to means testing. If badgeholders finds this to be limiting in reviewing projects for RetroPGF 3, the Collective may iterate on this in future rounds.

Call for Feedback

• Does the application form provide a smooth experience for applicants? Are there possible hurdles or unnecessary friction?
• Is the application form aligned with the review process of applications?
• Does the information provided by projects help badgeholders review their impact & profit?

[lefterisjp]

Hey thanks for the post.

As I have already mentioned few times in Twitter:

• https://twitter.com/LefterisJP/status/1704566169393401996
• https://twitter.com/LefterisJP/status/1706713742753366233

I find the fact that VC funding info is not requested but then donations and grants info are requested really bad. From my chat with some of the people working on this and from the post you say:

The reasoning behind this is that venture investment is made in exchange for ownership in a project and, therefore, should not be considered as profit.

And that this encourages something you guys called “impact investing”. So that investors who invest in public goods also get rewarded by retroPGF and are more encouraged to invest in public goods later.

---

I disagree with this view on things and find it rather broken. The fact I have to write this makes me really sad. I will try to explain with clear arguments what I believe is wrong. And will provide the points that are wrong from both perspectives. The perspective of a badge holder and that of an applicant to retroPGF.

As a badgeholder

I absolutely, most certainly 100% want to know if a project has raised VC money, how many rounds and in what terms. Why?

Because this shows me:

• Is it really a public good? Or a private good owned by its investors?
• How much money the project has in its war chest. I can compare with their impact and then see if they have need for more funding.
• I want to know who the money goes to. As transitively the investors own the projects and can control its direction. It’s not unusual for the same VC to be behind 100s of project. If all of them apply for retroPGF and in the end all that money goes to the same VC funds then this system is utterly broken.
• Impact is measured depending on the resources the projects has had to spend and depending on the size of the team. If the team has had a nice impact but that was mostly due to being subsidized for everything by a VC then this project is a lot less deserving of RPGF.

As an Applicant

For same reasons as above you should have to declare all investments into your person and project. What’s more if you don’t then you create a completely unfair and imbalanced system.

Let me explain why. RetroPGF, much like Gitcoin grants, at some point devolves into a popularity contest. The more well known a project is the more potential it has for some points allocation by the majority of voters/badgeholders just due to its name. Let’s take Safe as an example (notice: I invested in them so right now I am fighting against my own interests) of a VC funded project that has raised a lot of money ($100m) and also participated in retroPGF (round 2). Taking that into consideration then we have:

• Projects that are already well funded and have very big marketing teams, sponsor events and even themselves invest in other companies are known by everyone in the field. The fact that everybody knows them is backed by the VC funding they got since that’s what allows constant presence in events, marketing, sponsoring etc. That provides them with an unfair advantage over other projects that are virtualy unknown. Taking safe as an example, every badge holder will see it, say ah yes safe I know them. Let’s send some points there. If they knew safe has raised $100m while let’s say otterscan is just one guy, then they may consider twice before choosing safe over otterscan for RPGF.
• The comparison between grants/donations are on a totally different level and scale. Projects that raise VC funding gain a war chest of 10s-100s of millions of dolllars (on which they owe no taxes due to investment). Safe raised $100m. Donations and grants are denominated in a few thousand of $ if the project is lucky. And they owe taxes on them too. Say rotki’s last gitcoin round was $817 of matching, 40-45% of which is owed in taxes already. We are talking about a totally different scale.
• This is more personal/subjecive, but find it insulting and unfair to ask for detailed info on grants/donations but specifically exclude VC funding. This sends the absolutely wrong message about what kind of projects RPGF wants to support and it’s a big middle finger to people who try to build true public goods in a bootstrapped way.

[dmars300]

Hey @lefterisjp

I agree that transparency about VC funding and/or investments is essential for fair RPGF allocations. It’s key to know a project’s full financial situation, not just grants and donations. Resonate with most of your points made.
I believe it is important to modify the RPGF form to require details on VC backing as well.

[Griff]

I totally understand the whole Impact = Profit cycle intention. Bootsrapping that cycle is awesome and if we do it right, people will invest into startups that are going to make Impact, and they shouldn’t be punished for that especially since getting investment is not really a “Profit” for the organization… However, I have to agree with Lef.

This is my understanding of what we want with the RetroPGF cycle, it is beautiful… I rant and rave about it to anyone that will listen… but we aren’t there yet. I don’t know of any projects that sold investors a story about getting money from RetroPGF to pay out to the investors. We are still
in the bootstrapping phase, so the investments that will show up this round will mostly see RetroPGF as a bonus. That said, even after this flow is happening… we should still be made aware of the investment numbers.

As a badge holder, I want to simply ask you to sell us on the vision of RetroPGF so we can deprioritize the investment numbers, but don’t intentionally exclude this relevant info.

Understanding the investment #'s is important to understand the whole picture of a project. Explicitly excluding it feels like we are being treated like children that just couldn’t understand.

Last round the badgeholders actually passed around a spreadsheet to track the investment numbers they knew of… I assume we will do the same this round, so it might as well be included in the form. If we want the info we will make it happen.

It would be an easy change to just explicitly include investment #'s in the same question.

[ismailemin]

Firstly, I would like to address the issues we encounter when submitting applications:

I agree that the 800-character upper limit is a helpful factor for streamlining reviews, especially for badgeholders. However, it doesn’t make sense to hinder projects that wish to share more information. To address this, perhaps a system could be implemented where the initial 800 characters are displayed, and badgeholders who wish to see more details can click a ‘read more’ button, similar to how it works for longer tweets on Twitter.

I also believe that some metrics may not be measurable with numerical values. Therefore, I think it’s necessary to include mandatory non-numerical metrics.

I also believe that Lefteris’ point mentioned above is valid.

@lefterisjp

The comparison between grants/donations are on a totally different level and scale. Projects that raise VC funding gain a war chest of 10s-100s of millions of dolllars (on which they owe no taxes due to investment). Safe raised $100m. Donations and grants are denominated in a few thousand of $ if the project is lucky

@lefterisjp

The more well known a project is the more potential it has for some points allocation by the majority of voters/badgeholders just due to its name.

I don’t think a grant of $100-200K from RPGF would have a significant impact on a company like Safe, which has received millions in investments. Moreover, companies like Safe have substantial marketing budgets as a result of their investments, which they can use to promote themselves. Communities like ITU Blockchain, of which I am a member, do not have large marketing budgets. Therefore, this grant program should be made more equitable.

[Gonna.eth]

I think on the contrary 800 cha is fantastic. You can add a link to a doc or a website if you want to give more info. But this limit pushes everyone to tell the most relevant things about a project and avoid scattering.

[Griff]

I agree with gonna. The character limit is important given the fact that 150ish badgeholders need to digest sooooo many projects, keep it short and link out to another page if you want to set context.

There are already 288 submissions. I assume a bunch of them are just tests… but still we are looking at easily over 1000 projects that will submit.

Schema #114 - 0x76e9...e77cc

[CryptoReuMD]

@lefterisjp

• Is it really a public good? Or a private good owned by its investors?
• How much money the project has in its war chest. I can compare with their impact and then see if they have need for more funding.
• I want to know who the money goes to. As transitively the investors own the projects and can control its direction. It’s not unusual for the same VC to be behind 100s of project. If all of them apply for retroPGF and in the end all that money goes to the same VC funds then this system is utterly broken.
• Impact is measured depending on the resources the projects has had to spend and depending on the size of the team. If the team has had a nice impact but that was mostly due to being subsidized for everything by a VC then this project is a lot less deserving of RPGF.

-This is something that need to be implemented and some of the questions that need to be adressed with every decision and every project. “Public Goods” It’s now memetic since OP exploded the world, and now the code of conduct could be Wolfs in sheep clothing, projects that already been funded, and just looking for more voting power or more influence for their own projects.
We need to be careful, i’m here as an observant and a learner, but im part of ethics, and relevant projects that are related to public goods and i can’t manage that we are having a little trouble over here. But as a crypto dreamer, i’m sure that also the decisions needs to be merely centralized, becase we are not experts and we are learning.
I can help, or just comment haha. I love reading this.

[MinimalGravitas]

I’d like to ask a question about L1 infrastructure and how this could be placed into one of the 4 RPGF categories?

We’ve recently seen Blocknative’s MEV relay shut down, reducing the decentralization of this aspect of L1. Much like clients, relays don’t really have a source of income, but yet support the entire ecosystem and so Blocknative are unlikely to be the only one to wind down operations over time. The fewer there are the more vulnerable the network becomes to censorship.

I’d like to support these if possible with RPGF, but they don’t seem to fit neatly into any of the categories, all of which are, quite reasonably focused on Optimism, the OP stack, OP governance and other things starting with ‘OP’!

My argument would be that Optimism (and all other L2s) depend on being able to use Ethereum mainnet, and therefore anything that reduces the possibility of censorship on L1 is indirectly a benefit to us, and so worth supporting.

I’m not involved with any MEV relays or anything, and I’m not even especially knowledgeable about their economics or whatever, but one of the very small number has closed down and I’d like to try to reduce the risk of more following them!

If you have no idea what I’m rambling on about, then this episode of the Empire podcast was what helped me get my head round it.

Would it be reasonable for MEV relays to apply under the category ‘OP Stack’ perhaps?

[joanbp]

A couple of thoughts…

First

Being new here, I notice the amount of care and effort that has already been put into this process, and I really appreciate the openness to continuous experimentation and improvement!

VC funds and means testing

I agree with others in this thread that applicants should be asked to share information on any VC funds.

More generally, I find that at least a basic form of means testing is indeed (or should be) very relevant in the voting process.

While it is true that retroPGF is about retroactively supporting anyone who has already contributed to the creation of public goods, we do so with the explicit intention of ensuring future creation of more public goods.

• Public goods that are not backed by large VC funds are much more likely to not be initiated or continued without our support.

• Decentralization and diversity are very important public goods. Means testing will give us a better basis for judging the significance of choosing one project over another.

• Badge holders would still be free to support projects that are backed by VC funds. By asking applicants to declare themselves, we simply get a clearer picture of where any grant money will land, and what kind of difference it could make.

Measurement

I have been wondering about this illustration from the RetroPGF3 application guidelines:

A single metric is specified for each category.

However, as I understand it, the overall intention is that applicants should be free to draw attention to the kinds of metrics that they find relevant to their specific project. These metrics could be both qualitative and quantitative.

@system

While the Collective has defined broad impact categories for RetroPGF, projects often have a more detailed understanding of how to measure the impact of their work, and it is important to empower them to surface their impact to badgeholders as they see fit.

I suggest changing the header text ‘Measurement’ in the mentioned illustration to ‘Example measurement’ - to avoid giving a false impression to badge holders and potentially scare off applicants.

Also: In the guidelines text (below the illustration), it says:

“Each category has a success criteria associated with it. If you’re choosing a category, be sure to connect your impact description and metrics to the way your contributions helped drive towards that category’s success metric!”

I’m not sure if the “success criteria” is thought to be the short description of the category (eg. “Work that provided impact to end users in the Optimism Collective, or helped bring new end users into the collective”)? In any case, the phrasing “that category’s success metric” (singular) strongly suggests that the single example measurement is somehow special, primary and maybe even all-important. I suggest rephrasing this just slightly to something that better fits the intention of inviting people to highlight any concrete metrics that they might know of that is relevant to their project.

[chom]

I think we should be able to put images. An image is worth thousands of words. Mostly used to show statistics/examples of how our application looks like.

[Jonas]

The OP Stack category includes dependencies (e.g. Ethereum blockspace). Definitely reasonable for them to apply under the OP stack category.

[MinimalGravitas]

Perfect, thanks for the confirmation!

[Jonas]

Thank you for the feedback!
The responses have made it clear that badgeholders want the application form to include questions on the venture investments that applicants received.

The decision to not include this question in the application form was made because we didn’t consider venture investment as critical to the voting process in RetroPGF. This is explained further in an “Impact = Profit Framework” as part of the badgeholder manual. (Updated with link on Oct 31st)

As the application requirements can not be changed while the process is ongoing, it was not possible to address this feedback during the RetroPGF 3. This will be taken as important feedback for future RetroPGF experiments and will be included in the RetroPGF 3 retrospective.

In future rounds, we will share the application form for feedback before it is finalized.
We’ll be collecting your feedback throughout this entire round and will present a RetroPGF 3 retrospective at the end, like we did for RetroPGF 2.
Thanks for taking part in this continuous experiment, your input is a critical component in the evolution of round design!

[Michael]

I created a thread for this but realize I should have just posted it here.

Proposal:
Create a staking requirement of 10 OP to submit an application to RetroPGF 4. If the project receives any funding from that round, that stake is returned. If not, the stake is added to the overall RetroPGF pool.

Why?
Currently there is zero skin in the game to applicants and no cost to submitting a spam application. Adding a very small staking requirement (10 OP) should drastically improve the signal-to-noise ratio of applications and eliminate the low-effort spam.

I realize this isn’t a revolutionary idea but wanted to get it into writing to get feedback on implementation. It’s worth noting that EthGlobal does this for their hackathons.

[santicristobal]

I think this is a great idea and could significally improve the RetroPGF application process.

An alternative implementation of this idea would be using Kleros Curate, which I propose checking out.

Similarly to what Michael suggested, applicants would have to stake to be able to submit. The difference is that then other members of the Collective will be able to challenge those applications. If the challenge is accepted and the application ends up being rejected, the challenger receives the staked tokens.

While I like the idea of adding the lost staked tokens to the RetroPGF pool, I think Kleros mechanism could be a could experiment to increase community participation in the filtering process.

I was discussing this idea with a friend from Kleros who can probably outline a potential implementation much better than I did; he will probably hit the forum with more details soon.

[CryptoReuMD]

@Michael

Currently there is zero skin in the game to applicants and no cost to submitting a spam application. Adding a very small staking requirement (10 OP) should drastically improve the signal-to-noise ratio of applications and eliminate the low-effort spam.

Nice. I think it’s could be great. When we apply for a Hackaton we stake some tokens to prove our commitment and then this tokens are released. This is a problem to the community since off chain transaction or really low cost transaction could be spammed, so I’m truly interested and could be a very good approach.

[joanbp]

@Michael

Proposal:
Create a staking requirement of 10 OP to submit an application to RetroPGF 4. If the project receives any funding from that round, that stake is returned. If not, the stake is added to the overall RetroPGF pool.

I think it would be better to only withhold the stake if the application is considered spam.

There are no guarantees that all non-spam applicants will receive funding. E.g. some may be completely legitimate, but their impact deemed too low (given the amount of funding to be distributed and the high quality of other applications). They should not be punished for applying.

But I agree that punishing actual spam would likely reduce the amount of it significantly.

And I like your idea of keeping the stake small - it should not scare legit applicants away, just discourage spam.

One thing to consider, though, is that it puts a bit more pressure on the initial reviewers, knowing that the projects they reject will be economically punished like this. Then again, this may just keep things real for everyone.

[Pumbi]

Completely in favor of this proposal.

It is not a difficult requirement to meet, and greatly discourages spam applications.

@joanbp

There are no guarantees that all non-spam applicants will receive funding. E.g. some may be completely legitimate, but their impact deemed too low (given the amount of funding to be distributed and the high quality of other applications). They should not be punished for applying.

This point also seems very valid to me. Being a legitimate application, in the case of not receiving funding, projects/individuals should not suffer a penalty for that.

[chuygarcia.eth]

@santicristobal

An alternative implementation of this idea would be using Kleros Curate

Involving Kleros in anything is a horrible idea. Kleros is a broken system where incentives work towards the presence of disputes, instead of their solutions.

There’s plenty of evidence on how one of its co-founders has attacked the protocol for his own benefit, something that has recently been also confirmed by a whistleblower.

Kleros shady comms lead @0xjean is also very close to @CryptoChica who has been involved in a number of cases of corruption, conflicts of interest and other serious accusations where no explanation has been provided by her organizations SEED Latam and ETH Kipu to the Optimism Collective. No wonder why organizations that have been benefited by them keep trying to push Kleros wherever they have influence.