AI Copilots for Optimism RetroPGF

[akrtws]

Introduction

We are @rororowyourboat and @akrtws from the Token Engineering Academy’s cadCAD GPT initiative. Over the last months, we’ve developed an open-source framework to support token system simulations with LLM agents (Large Language Model agents).

More about cadCAD GPT:
Github: GitHub - TE-Academy/cadCAD-GPT
Mirror: Hello, cadCAD GPT! — Token Engineering Academy
Demo session: https://x.com/tokengineering/status/1730516745654251787?s=20

While working on cadCAD GPT, we realized that there’s huge potential to utilize such agents for better, faster, and more data-driven decision-making in DAOs.
The major benefit of such agents is that they allow us to interact with system models, data, and code in human language.
This way, it enables DAO decision-makers from multiple backgrounds and perspectives to run data analysis, and even the most complex simulations without writing code.

Demo MVP

To show the potential, here’s a short video, demoing RetroPGF GPT.

OPRetroPGF Demo Video

In the video, we present use cases to give you a picture of how powerful AI agents are, and how they can support RetroPGF:

• analyze RetroPGF voting outcomes and compare metrics (like discussed here, great comparison by @Pr0 )
• create new data sets, and charts, summarize and export for further usage
• develop new metrics with AI support
• assess results from a badge holder point of view (my voting vs. total badge holder voting)

This is just a small sample of what AI agents can do for RetroPGF.

There’s much more:

• During RetroPGF rounds: support badge holders in evaluating grant applications
• After RetroPGF rounds: find flaws or vulnerabilities in the voting or distribution process
• Improving RetroPGF: compare different voting mechanisms and evaluate pros/cons (like discussed here, awesome analysis @amy @ccerv1 !)

A suite of AI agents for Optimism Retro PGF

Note that the agents in the demo video shared are made using GPT Builder, a closed-sourced tool by OpenAI. It makes building MVPs super simple and efficient. We’ve equipped our GPT in the demo with RetroPGF raw data available here, and only made some minor tweaks, like adding a fictional badge holder voting ballot.

However, we aim to build a suite of LLM agents customized for Optimism RetroPGF:

• Equipped with an ever-growing set of tools and information that are particularly relevant for Optimism RetroPGF
• Open-source as much as we can, and as much as the Optimism Collective is willing to open-source (we understand that some information or tools should not be public, like the voting behavior of individual badge holders)
• No OpenAI lock-in. Modular framework to integrate LLMs, so that we can either use OpenAI (most powerful today), or switch to alternative, open-source large language models like LLaMA2 or Falcon in the future (yey!)
• Modular framework on the tool side, so that more sophisticated metrics and models developed by Impact Data Scientists, can be integrated as Python plug-ins
• Inform the RetroPGF data creation/collection/cleaning/pre-structuring process so that raw data (such as project applications for OP RetroPGF) is optimized for LLM agent consumption
• Train users, such as badge holders, to use and make the most of AI agent support

Next step: Building first tools for badge holders supporting RetroPGF round 4

AI-powered DAO governance has enormous potential. Currently, several DAOs are exploring the application of AI to their decision-making process. With RetroPGF, Optimism would be definitely at the cutting edge. It is a true moonshot!

But let’s be honest, AI-powered RetroPGF is a huge endeavor, too.

So let’s start with an actual use case, and provide value: We’ll build the first LLM agents to support badge holders in RetroPGF round 4.

Our main questions:
What’s the most pressing need for assessing RetroPGF applications for badge holders? How to best integrate voting improvements discussed in the community?

We’d like to talk to the OP collective to put together a scope and requirements to ensure problem-solution fit. If it’s a fit, we’ll submit a mission application in Season 6.

If you are a badge holder, and interested in becoming an alpha user, drop us a line!

If you are a data scientist and want to make your analysis accessible through AI agents, contact us!

Here’s a Calendly link to book an onboarding call for alpha users / data scientists: Calendly - TE Academy

Also @Jonas , we need input regarding raw data access in round 4, who to best talk to?

And everyone, if you have any questions regarding this proposal, shoot away, we’ll be happy to discuss!

Thank you!

[joanbp]

This is amazing, and I would love to be one of your badgeholder alpha users.

I think what you demonstrate in your video is a wonderful way to use AI, leaving the responsibility with the human.

It would be really cool to see the kinds of questions that a group of badgeholders would think to come up with, and the AI’s answers. Also, I absolutely loved that you pointed out the need to also ask the AI about its sources of data and its calculations, so as to make sure that we really understand the data we are looking at and can reproduce them.

[akrtws]

Hi @joanbp - great, welcome on board as alpha tester! We’ll get back with more details in the next days.

I think what you demonstrate in your video is a wonderful way to use AI, leaving the responsibility with the human.

That’s the idea: it’s the Optimism Collective to decide which projects are legitimate, and valuable. Still, machine intelligence can add a lot of value in digging through data, finding evidence and getting the full picture.

[akrtws]

Adding a clarification:
we don’t aim to participate in Season 5 with a mission application. Instead, we’ll scope out application cases and build first MVPs with badgeholders, and if it’s a fit, come back with a mission application in Season #6. tbd.

We’ve edited our original post accordingly.

[latruite.eth]

Just saw the demo – super cool stuff, especially the part where we can see how GPT thinks things through. Quick question: have you guys figured out a way to feed in the “declared impact” data from Round 3 applicants? That would of course be super useful…

I’m not a badgeholder, but I would love to use it for a deep dive into Round 3’s results, maybe focusing on a specific project category. (Want to see how the declared impacts line up with the results ?)Hit me up if there’s a way to get involved in the Beta test.

[akrtws]

Hi latruite.eth,

thanks for your feedback!

Quick question: have you guys figured out a way to feed in the “declared impact” data from Round 3 applicants? That would of course be super useful…

Here, you refer to the Impact Information & Metrics provided by projects during the RetroPGF application process, right?
Technically, including this data is absolutely doable if we can get access to a data dump from Retro PGF → @Jonas would this be arrangeable? I have not been able to find this on the public Github.

a deep dive into Round 3’s results, maybe focusing on a specific project category. (Want to see how the declared impacts line up with the results ?

Mapping declared impact to funding received is one of the most interesting questions in RetroPGF, IMHO it’s the magic formula, since it’s a challenge to translate the variety of metrics declared to a coherent RetroPGF funding result, perceived as fair to the Collective and projects who applied.
See discussion on Quantifying Every Projects Impact as an OP Amount
@griff or The Role of VC Funding in RetroPGF

More thoughts (summarizing mostly what have been discussed in other RetroPGF3 feedback threads already):

• running this analysis is relevant to understand past rounds, and could reveal super valuable insights for designing future rounds
• Can/should declared impact metrics be verifyable and/or verified in the voting process? How?
• Metrics standardization? Can and should it be a goal to standardize impact metrics that count for RetroPGF, such as VC Funding received or Sequencer Revenue Created @alexcutlerdoteth
• Category Impact Metrics? Develop and define meaningful impact metrics per category, like number of active users in category End User UX, or #stars on Github in Developer Ecosystem to enable data-driven project comparison - which would be particularly valuable in case projects should be ranked in round 4.

Let’s explore these questions with GPT support!

[akrtws]

@latruite.eth @joanbp and everyone:
just updated our original post and added a link to book an onboarding call to RetroPGF GPT - just go ahead and reserve a slot, see you there!

[latruite.eth]

Yop, would be awesome to get access to the impact data declared by projects in their applications. (Gotta be careful with the data integrity, of course.)
I’m fairly new to data analysis, but eager to explore I’m planning to do an article on a specific subgroup of projects: the .End User Experience Adoption: Evangelism & User Onboarding category (113 projects/individual applications).
Already booked a timeslot for an onboarding call , thx!

[ccerv1]

Very cool! Just booked a time to speak with you on 26 Feb

[ehsanomics]

This is super cool! Thanks for building it @rororowyourboat & @akrtws !!

[akrtws]

Update:

We’ve equipped our prototype with project data (information projects submitted in round 3 applications).

RetroPGF GPT has now access to:

Project Data: This dataset provides detailed information about each project, including their bio, impact category, display name, website URL, applicant and applicant type, profile, impact and contribution descriptions, contribution links, impact metrics, funding sources, lists, ID, preliminary result, and report reason, along with their inclusion status in ballots.
RPGF3 Results: This dataset contains information about each project’s name, the number of ballots it received, the median amount of funding proposed, whether the quorum was reached, the amount of OP received, and the project profile link.
Anonymized Project Votes: This dataset includes the votes that each project received from all voters, including specific votes cast by an identifiable voter. NaN values indicate where a vote was not cast.

You can now:

Explore projects:

• explore project application data
• filter projects, develop and reason about filter criteria and parameters

Explore voting results:

• analyse the voting outcome (per round or across voting rounds)
• explore through different lenses, find correlations

Analyse your own voting preferences

• compare your own ballot with overall voting results
• explore and understand your voting preferences, let GPT find patterns

Plus you can create new lists, create charts, and export findings!

Want to try it out?
Book a badge holder onboarding call, and get access!

@ccerv1 feel free to book the test session via the same link.

[curvewars]

This is a great idea to use LLM’s to summarize the long proposals on the forums and also add some automation to unveil insights about most active users. This can also be used to potentially do data analytics on how grant proposal winners are spending funds.

Happy to be a early beta tester

[akrtws]

Hi @curvewars thanks for your comment!

This can also be used to potentially do data analytics on how grant proposal winners are spending funds.

Yes technically we can e.g. track and analyse the journey of OP tokens distributed via RetroPGF.

I guess the underlying question is “Have the funds been put to good use”? – which is a sligthly different question, since it requires more than tracking how tokens move from wallet to wallet. Who are the owner of these wallets? Why did funds move? (make payments? compensate contributors? manage a treasury to secure the value?)

That said, for measuring “put funds to good use” I tend to prefer measuring the output of a team, and the progress they made, rather than the funds itself.

You might have more ideas! Feel free to book an onboarding call.

[akrtws]

Hi all,

here’s an update on our work for AI Copilots for Optimism RetroPGF.

First, thank you to @joanbp @lee0007 @latruite.eth @ccerv1 @launamu @dmars300 and @brichis for testing our prototypes and providing feedback!

DappCon 2024 Talk
At Berlin Blockchain Week in May, we summarized our work on AI Copilots in a talk, and mapped out the potential of LLMs in the context of token systems and Retro Funding (RetroPGF respectively).

• How LLMs support token system design
• Why the system’s perspective is relevant for the development of Retro Funding
• How LLMs can support educated, data-driven decisions and maintain a diversity of perspectives in evaluation for Retro Funding

Some screenshots are below.
Watch the recording of the DappCon 2024 talk!

What’s next?
Since April, we have been working on TE Academy’s Retro Funding program, the first Fellowship grant prize. To select the winner, we’ll run a community voting in June, assigning voting power to voters according to their reputation.
Developing this voting mechanism is part of our educational program that we offer thanks to Retro Funding by Optimism! We develop the voting mechanism with our students.
Learn more about this program here.

Some exciting questions to explore:

• Can LLMs support Voting System Simulations, and make experiments more efficient and faster
• Can LLMs come up with design variants, that we haven’t thought of
• Can LLMs help to communicate the results, so that decisions on a voting design can supported by LLMs?

More updates in July!