Context
Reputation scores for Developers and OP stack GitHub repos are used for metrics in Retro 5. They are based on calculating recursive trust across the Github developer activity graph, powered by OpenRank and OSO. This ranking is seeded by OP Stack repos. This means Github repos and developers are scored based on recursive trust signals, originating from these seed trust repos.
A graph of relevant GitHub event data from Open Source Observer, covering more than 2,000 organizations and over 30,000 code repositories, was used in generating these scores. The graph has two kinds of peers: GitHub users and repositories. We then used a variation of EigenTrust — the Hubs and Authorities algorithm, which computes rankings on an asymmetric bipartite trust graph. The following visualization shows the repos and developers as nodes, and how they are forming an integral web.
We believe introducing reputation metrics is important because 1) voters need better signals to know if a repo is contributing to OP stack and to what degree is that contribution; 2) voters need signals to know if a repo has been interacted by reputable developers, which adds to an underlying endorsement of the value of the repos.
For Retro 5, a first version of the metrics is generated by OpenRank. This is just the beginning of experimentation with reputation based metrics. We look forward to your feedback and input in making the scores and rankings more relevant and meaningful.
Now you can create your own Rankings and Reputation scores
We’re excited to launch a client for anyone to create their own version of reputation scores. Check it out here: https://devrankings.openrank.com/
It allows any user to understand how the reputation scores are computed, and create their own custom rankings. The cool part about this is that the OpenRank compute is happening client side.
Step 1: Select ranking criteria - which is basically recursive trust arcs that are taken into consideration to quantify interactions between nodes in a trust graph. Here we set as default the same criteria as used in Retro 5. In this version, you don’t need to change or configure this.
Step 2: Input any Github developer handle or repo as Seed Trust - this input is used as Seed Trust for the recursive trust graph. Trust propagates from these repos through interactions outlined in step 1, to the rest of the repos and developers, based on one’s proximity to selected Seed Trust. In Retro 5 metrics, OP Stack core repos were used as Seed Trust.
Step 3: Select a seed strength (between 0 and 1) - this alpha value (α) is a parameter in EigenTrust which determines the degree of influence of the seed peers in deciding the reputation scores of peers in the graph. Alpha value is normally chosen between 0 and 0.5 and the higher the alpha the stronger the influence of seed peers, meaning the trust arc from seed peers to other peers will matter more for a peer’s relative reputation.
Finally, you can generate and download the your customized reputation scores.
Access the scores via BigQuery tables: openrank-data.devrank_2024{0627,0910}.{user,repo}_scores
- There are two versions of the scores used in Retro 5, and you can access it via OSO data portal too:
openrank-data.devrank_20240627has the scores used for Optimism Retro 5 guest voter selection. It uses GitHub activities up until June 27th, 2024 as the input data.openrank-data.devrank_20240910has the scores used for Optimism Retro 5 metrics. It uses GitHub activities up until September 10th, 2024 as the input data.
- For historic reasons, the two score sets are not on the same scale. In order to compare, multiply the scores in the
openrank-data.devrank_20240627dataset by 2.25014877694904 (user score) and 1.49998254193340 (repo score).
Learn more about the rationale of the scores here.
The scores will be available on Open Source Observer data portal too.