Retro Funding Season 7 Governance Process

Retro Funding Missions đź”´
,
5

Hi Mel
I appreciate your post and, as the data lead for OSO, will say that I share the same goal as you wrt a competitive landscape for evaluation algos. We have no desire or intention to be a sole provider here, and the fact that we are for S7 is mainly a reflection of the huge lift to get the first versions of these algos up and running.

One common source of confusion that I want to address is that the Foundation requires “models” at various levels:

  1. ETL/Pipeline, ie, to clean and connect various data sources. For example, we want to query: “How much gas did the onchain builders who import all npm packages owned by project X contribute between Feb 1 and Feb 28”. This requires building models to connect different datasets maintained by different orgs.

  2. Defining metrics. For example, we might create a metric called downstream_gas_30_day_total which allows you to look up a project and date, and get its total Superchain gas contribution over the preceding 30 days and another called downstream_gas_30_day_net_change which looks at the absolute difference for the current period relative to the preceding 30 days. This requires getting feedback from various stakeholders on what is important to measure (and then implementing it).

  3. Combining metrics into evaluation algorithms. Basically, some way of weighting the various project-level metrics towards some overall strategy. For example, a growth-focused strategy might weight a metric like downstream_gas_30_day_net_change more heavily than other metrics. This requires a lot of data science!

My impression from your post is that you are mostly concerned about level (3), perhaps (2), but not so much (1). If so, then I think we are on the same page!

Level (1) is the part that OSO is primarily focused on. This is a data engineering problem much more than a data science problem.

There reality is there are several really big datasets that need to be connected in order to get to level (2) and derive metrics for the upcoming devtooling and onchain builder rounds. FYI, OSO is not the maintainer of any of these datasets. The maintainers are:

  • Superchain data → OP Labs
  • TVL data → DefiLlama
  • GitHub data → gharchive
  • Software dependency data → deps.dev
  • Project registry → OP Atlas (via EAS)

(You can get more info here.)

Again, the connecting is OSO’s primary focus. We are not aware of any other teams doing the connecting in a public way. That said, as @Jonas mentioned, the pipeline is completely open source – even the infra is fully open. (View any job in our pipeline here; code is permissively licensed if you want to fork it.)

Once all the data is connected, then data scientists can have fun creating different metrics and creating different eval algos on top of whatever metrics / features they want to use (levels 2 and 3). We want to facilitate this as much as possible. You can grab the data however you like. (Of course, as a community-led project, we’d love if you shared back your best models as a PR to our code base, but there’s no requirement to.)

As of today (Jan 27) we are still hard at work with OP Labs, Foundation, Agora, and other teams trying to connect all the necessary datasets. Once the round opens, there will be a lot of testing to ensure that all the necessary event data is being pulled for each project artifact and underlying metrics are being calculated correctly. It would make zero sense to have data scientists competing to build eval algos until the underlying source data is locked. That’s the work happening now and through the end of this first season (July).

Hopefully this addresses your primary concern, but just to recap:

  • Source data to build models; 100% public, we don’t maintain it
  • Pipeline to connect models; 100% public and open source; we maintain this but you can fork it
  • Metrics implementation; anyone can contribute via PR (or share in their own provided they are public and reproducible from the source data)
  • Eval algos; goal is to open this up and have lots of competition. Also, the algos will be maintained in an OP repo (not an OSO repo).

Finally, I will end by reiterating that we as OSO have no desire to be in the eval algo game long-term AND would love for this to be competitive. Here is an example of a competitive process that we are facilitating for DeepFunding (albeit on a much smaller amount of data).

6 Likes