Introducing Open Source Observer: measuring impact of OSS contributions to the OP Superchain

[ccerv1]

With RPGF3 voting kicking off today, we are excited to present a new public good for the Optimism Community: Open Source Observer (OSO)

tl;dr-

• We’ve created a huge directory of 900+ OSS projects and nearly 20K off-/onchain artifacts
• It includes every project that applied for RPGF3 with at least a GitHub repo – over 300!
• For every project, we’ve indexed a suite of metrics … here’s a spreadsheet you can start playing with
• There’s even more data up on our website () including project deep dives and (soon™) sortable metrics tables
• We’ll be using the data to generate voting lists for badgeholders based on quantitative impact metrics
• Jump in this group chat if you have issues, need support, or want API access!

Intro

OSO aims to create better visibility on the impact of different types of open source software contributions to the health of the OP Superchain.

In the long run, we want OSO to be a source of verifiable proof that “impact = profit”, at least for impact that is traceable to open source software contributions. And we want that data to be a free and easy-to-use source of business network intelligence for badgeholders, citizens, and builders.

What we’ve done so far

We started this project about 3 months ago. Here are some of the highlights:

• Created a registry of projects that are contributing to Optimism
• Linked projects to “artifacts” such as git repos, smart contracts, and npm packages
• Indexed event data about those artifacts such as commits, contract interactions, and downloads
• Defined initial activity and impact metrics such as monthly active developers, onchain users, and package dependents
• Released an alpha version of a GraphQL API and dashboarding interface for exploring the data
• Dogfooded the data to generate some analysis on the OSS projects applying for RPGF3 and to help power voting lists for badgeholders

Registry

You can view the registry of 900+ OSS projects and nearly 20,000 artifacts here. Our RPGF3 coverage includes 300+ projects, 5K+ repos, 4K+ contracts, and 50 npm libraries.

The above should include all of the OSS projects in RPGF3 as well as any contribution links in their applications that specified a GitHub repo, OP Mainnet contract address, or npm package. If we missed anything (which we most certainly did), please open a PR or send us a message.

Indexers

We built a number of indexers to capture event data from these artifacts and normalize the data. You can check them out in the OSO monorepo. Most of the indexers are able to perform daily updates as well as historic backfills even to pre-Ethereum days. We’ll be indexing other chains and package managers soon … and are very eager to partner with other projects that have useful data to layer on. If you have ideas or notice bugs, please open an issue or send us a message.

Using OSO Data

Finally, we’re proud to share some different ways of accessing and composing with this data:

• Data Snapshot: we created some initial metrics for 300+ RPGF3 projects and dropped them into a google sheet. Have fun with it!
• API: We have a GraphQL API that you can use to make queries directly. Send us a message if you want access – we don’t have much in the way of docs but are happy to provide as much support as we can manage at this time.
• Dashboard: We have a dashboard interface running on top of the API available at opensource.observer. You can analyze cohorts of projects or deep dive on specific project metrics. We released this ahead of schedule, but it’s still got some performance issues and obvious bugs we’re squashing.
• Lists: We have an insights repo for generating voting lists. We’ve populated it with some initial strategies, like ranking projects based on the number of stars per active contributor or the time delta between the project’s first commit and its first transaction on OP Mainnet. We’ll be patching gaps in the data and refining these lists over the next week.

Screenshots

From Velodrome’s page - sample onchain metrics:

From IPFS’s page - sample GitHub metrics:

From our own page - sample NPM metrics:

Sustainability

Ray, Reuven and I began this work while employed at Protocol Labs working on hypercerts. We left in October to work on this full-time. Our new company is called Kariba Labs. OSO is the first thing we’re releasing.

We are taking an “Open Source, Open Data, Open Infra” approach to what we build. That means all our code is developed using permissive licenses (e.g. MIT/Apache 2.0). All collected and processed data will be openly shared with the community (to the extent allowed by terms of service). And we will open up our infrastructure for anyone to contribute or build upon our existing infrastructure at cost.

In terms of funding, we received a Builders Grant in July and won the “Best App for RetroPGF, Public Goods, or Measuring Impact” track at Superhack in August for an early implementation of OSS Directory. We also secured some seed funding from the Protocol Labs Network.

Community

If you want to connect, discuss an issue, or follow updates, join our Telegram chat here. We just created it, so there’s no one in it at the time of writing but we’d love to see you there!

[Xofee]

I wanted to quickly say how impressive the Open Source Observer project is. It’s great to see such support for open-source projects. Well done!

On that note, I have a project that might fit within your scope. I’ve developed a random number generating machine used for the KZG ceremony, which is part of my RPGF3 presentation. The GitHub repository for this project includes all the necessary code and even 3D files for its construction. It’s an open-source endeavor that could potentially be beneficial to anyone needing to conduct a trusted setup in the future.

Here’s the link to the GitHub repo: GitHub - KZG-Marble-Machine

I thought it might be a useful addition to your repository, should it align with your criteria. Thank you for your consideration and keep up the fantastic work!

[ccerv1]

Thank you for the kind words Cool project! I just merged a PR to add your project to the OSS Directory.

PSA: feel to make a PR if you want to add / update your project info too!

[s_ben]

This is very interesting! This general approach is imo the most promising for measuring impact of OSS, and glad to see a serious crack at it.

Do you have thoughts on tea.xyz, which appears to be attempting something similar (though not OS yet iirc)?

Tea.xyz: A New Way to Fund Open Source Software with Max Howell

[ccerv1]

Yes! We’re big fans of tea.xyz. Drips is another amazing project working on dependency funding. Integrating with both protocols is on the roadmap!

[s_ben]

Nice! It’s happening…

[InbestProgram]

Hello! let me start by saying that its amazing all the work you guys are doing with this observer, and the name is the best congrats!

I wanted to point out that our project is not showing up on the observer, i just realized this and its a bit late on the campaign, but im Shure that if we can make it so that InBest Program is recognized by the observer it would greatly improve awareness around our project.

Since we are on the education, community, and onboarding side, there isn’t much we do on chain nor on GitHub, that may be the reason, that being said we do have a GitHub but the handle is (wkss1) wkss1 (Alberto | InBest Program) · GitHub

That being said we are growing and now that we know, we will deff be doing more things on chain. thank for your work.

We are a spanish free community with hundreds of tutorials from what is a Blockchain or a L2, whats OPmainnet whats the superchain and OP Stack, how to bridge, how to how to run nodes, whats a DAO, what is gov, how to vote/delegate and much more, with a channel with over 1M imppressions and 80K Views on the last 8 months alone. InBest | Instagram, TikTok | Linktree

[ccerv1]

We wanted to provide an end of year update on OSO. There are three parts:

1. Our evolution from a small project on a large team to a full-time dedicated effort
2. Completion of our Builder Grant milestones
3. Retro on how OSO performed during RetroPGF 3

1. OSO’s evolution from a small project on a large team to a full-time dedicated effort

In our Builder Grant, we wrote:

It’s important to note that OS Observer is not a standalone entity. It’s a project currently being incubated by the Hypercerts Foundation and the Network Goods team at Protocol Labs. Therefore, we refer to “founders” in this context as the team who came up with the idea for OS Observer and who have been the primary internal champions for this project.

This was written in June 2023. A month later, we “nucleated” (spun out) from Protocol Labs.

We received a small nucleation package from Protocol Labs to put towards some of our startup costs and we launched an independent entity called Kariba. Open Source Observer is the first of hopefully various things we build together.

So … same mission, same name, same people … new home … and hopefully, more focus, more freedom, more impact!

2. Completion of our Builder Grant milestones

Here’s our proof of work for each of our Build Granter milestones that were due by 1 December 2023.

Project name	Milestone Type	Milestone	Proof of Work	Status
OS Observer	Benchmark	Data indexers completed for Github, package managers, dependencies, and blockchain data (eg, Etherscan, TheGraph, custom indexers)	See the OSO monorepo and check out the indexers, all run via GitHub Actions . Currently includes onchain, GitHub and npm source data.	Complete
OS Observer	Benchmark	Release of first impact ROI metrics, combining on-/off-chain data for any project or team building on Optimism	We created an insights repo to house our impact data science work and have been published various reports. For example, here is our report on 8 ecosystem impact vectors.	Complete
OS Observer	Benchmark	Over 500 projects with complete profiles (ie, linked Github organizations and wallet/contract addresses)	The easiest place to view all the projects were tracking is also on our GitHub in the OSS Directory repo. We blew through the goal of 500 and currently have over 1000!	Complete
OS Observer	Benchmark	V1 of interactive dashboard and impact evaluation interface released to the public [stretch goal]	The V1 shipped in early November, well ahead of schedule. See opensource.observer	Complete
OS Observer	Critical	Release of public API that can generate Lists for RPGF3 impact evaluation	We ended up serving static datasets in the form of speadsheets rather than an API. This seemed more useful given the composition of badgeholders and what we learned about voting strategies from our user interviews.	Complete
OS Observer	Critical	At least 15 unique Lists generated on EAS and shared with badgeholders	We shipped a total of 9 Lists include one List that was a roll-up of 8 Lists. Our insights repo contains a number of open source List experiments too. We did a retro on our lists here.	Complete

3. Retro on how OSO performed during RetroPGF 3

Here is a copy-pasta of our team retro on how things went.

• Given we were a new project with no website or brand until RetroPGF started, we are very happy with our first impression. We generated significant awareness and pushed data-driven analysis everywhere we could. The value add of this nature of work appears to be recognized by a majority of badgeholders and projects.
• We built a very powerful indexer, but our initial data warehousing infrastructure was not well-equipped for what we needed to do. This prompted a migration to a new data warehouse that should be more flexible and performant than what we built previously. As result, we dropped the ball on the Collections View feature.
• We learned that most of the power comes from being able to do custom analysis (eg, what Carl did with his own queries and python notebooks). As result, we want to prioritize getting a data collective up and running that can receive API access and cost-share some of the infra bills.
• RetroPGF has some requirements and mechanics that are challenging for OSO to accommodate. For example, we don’t have any reporting on non-OSS projects (~50% of all projects). What we view as a single project may choose to represent in RetroPGF as more than one projects (eg, Gitcoin vs Gitcoin Passport). A site like growthepie.xyz is better for “adding additional comparable fields to the RetroPGF set of projects”, although it has no concept of a “project” beyond how it’s defined during RetroPGF. A lot of integrations and collaborations to explore!
• The amount of data cleaning work was huge. Getting GitHub and package data is fairly manageable, but we need a better system for tracking contract deployments. This is something we need to discuss with the OP Labs team!
• No one has any clue how to think about impact measurement. This is exciting because it’s a frontier problem but also means we can’t just ask people “what do you want to see in this box”. There’s a lot of discovery and product testing work ahead of us!

---

Tagging Grant Council members @Gonna.eth @jackanorak @kaereste who reviewed our Builder Grant for visibility and welcome any feedback or comments