AI Copilots for Optimism RetroPGF

Introduction

We are @rororowyourboat and @akrtws from the Token Engineering Academy’s cadCAD GPT initiative. Over the last months, we’ve developed an open-source framework to support token system simulations with LLM agents (Large Language Model agents).

More about cadCAD GPT:
Github: GitHub - TE-Academy/cadCAD-GPT
Mirror: Hello, cadCAD GPT! — Token Engineering Academy
Demo session: https://x.com/tokengineering/status/1730516745654251787?s=20

While working on cadCAD GPT, we realized that there’s huge potential to utilize such agents for better, faster, and more data-driven decision-making in DAOs.
The major benefit of such agents is that they allow us to interact with system models, data, and code in human language.
This way, it enables DAO decision-makers from multiple backgrounds and perspectives to run data analysis, and even the most complex simulations without writing code.

Demo MVP

To show the potential, here’s a short video, demoing RetroPGF GPT.

In the video, we present use cases to give you a picture of how powerful AI agents are, and how they can support RetroPGF:

  • analyze RetroPGF voting outcomes and compare metrics (like discussed here, great comparison by @Pr0 )
  • create new data sets, and charts, summarize and export for further usage
  • develop new metrics with AI support
  • assess results from a badge holder point of view (my voting vs. total badge holder voting)

This is just a small sample of what AI agents can do for RetroPGF.

There’s much more:

  • During RetroPGF rounds: support badge holders in evaluating grant applications
  • After RetroPGF rounds: find flaws or vulnerabilities in the voting or distribution process
  • Improving RetroPGF: compare different voting mechanisms and evaluate pros/cons (like discussed here, awesome analysis @amy @ccerv1 !)

A suite of AI agents for Optimism Retro PGF

Note that the agents in the demo video shared are made using GPT Builder, a closed-sourced tool by OpenAI. It makes building MVPs super simple and efficient. We’ve equipped our GPT in the demo with RetroPGF raw data available here, and only made some minor tweaks, like adding a fictional badge holder voting ballot.

However, we aim to build a suite of LLM agents customized for Optimism RetroPGF:

  • Equipped with an ever-growing set of tools and information that are particularly relevant for Optimism RetroPGF
  • Open-source as much as we can, and as much as the Optimism Collective is willing to open-source (we understand that some information or tools should not be public, like the voting behavior of individual badge holders)
  • No OpenAI lock-in. Modular framework to integrate LLMs, so that we can either use OpenAI (most powerful today), or switch to alternative, open-source large language models like LLaMA2 or Falcon in the future (yey!)
  • Modular framework on the tool side, so that more sophisticated metrics and models developed by Impact Data Scientists, can be integrated as Python plug-ins
  • Inform the RetroPGF data creation/collection/cleaning/pre-structuring process so that raw data (such as project applications for OP RetroPGF) is optimized for LLM agent consumption
  • Train users, such as badge holders, to use and make the most of AI agent support

Next step: Building first tools for badge holders supporting RetroPGF round 4

AI-powered DAO governance has enormous potential. Currently, several DAOs are exploring the application of AI to their decision-making process. With RetroPGF, Optimism would be definitely at the cutting edge. It is a true moonshot!

But let’s be honest, AI-powered RetroPGF is a huge endeavor, too.

So let’s start with an actual use case, and provide value: We’ll build the first LLM agents to support badge holders in RetroPGF round 4.

Our main questions:
What’s the most pressing need for assessing RetroPGF applications for badge holders? How to best integrate voting improvements discussed in the community?

We’d like to talk to the OP collective to put together a scope and requirements to ensure problem-solution fit. If it’s a fit, we’ll submit a mission application in Season 6.

If you are a badge holder, and interested in becoming an alpha user, drop us a line!

If you are a data scientist and want to make your analysis accessible through AI agents, contact us!

:point_right:t5: Here’s a Calendly link to book an onboarding call for alpha users / data scientists: Calendly - TE Academy

Also @Jonas , we need input regarding raw data access in round 4, who to best talk to?

And everyone, if you have any questions regarding this proposal, shoot away, we’ll be happy to discuss!

Thank you!

8 Likes

This is amazing, and I would love to be one of your badgeholder alpha users.

I think what you demonstrate in your video is a wonderful way to use AI, leaving the responsibility with the human.

It would be really cool to see the kinds of questions that a group of badgeholders would think to come up with, and the AI’s answers. Also, I absolutely loved that you pointed out the need to also ask the AI about its sources of data and its calculations, so as to make sure that we really understand the data we are looking at and can reproduce them.

4 Likes

Hi @joanbp - great, welcome on board as alpha tester! We’ll get back with more details in the next days.

I think what you demonstrate in your video is a wonderful way to use AI, leaving the responsibility with the human.

That’s the idea: it’s the Optimism Collective to decide which projects are legitimate, and valuable. Still, machine intelligence can add a lot of value in digging through data, finding evidence and getting the full picture.

3 Likes

Adding a clarification:
we don’t aim to participate in Season 5 with a mission application. Instead, we’ll scope out application cases and build first MVPs with badgeholders, and if it’s a fit, come back with a mission application in Season #6. tbd.

We’ve edited our original post accordingly.

3 Likes

Just saw the demo – super cool stuff, especially the part where we can see how GPT thinks things through. Quick question: have you guys figured out a way to feed in the “declared impact” data from Round 3 applicants? That would of course be super useful…

I’m not a badgeholder, but I would love to use it for a deep dive into Round 3’s results, maybe focusing on a specific project category. (Want to see how the declared impacts line up with the results ?)Hit me up if there’s a way to get involved in the Beta test. :saluting_face:

2 Likes

Hi latruite.eth,

thanks for your feedback!

Quick question: have you guys figured out a way to feed in the “declared impact” data from Round 3 applicants? That would of course be super useful…

Here, you refer to the Impact Information & Metrics provided by projects during the RetroPGF application process, right?
Technically, including this data is absolutely doable if we can get access to a data dump from Retro PGF@Jonas would this be arrangeable? I have not been able to find this on the public Github.

a deep dive into Round 3’s results, maybe focusing on a specific project category. (Want to see how the declared impacts line up with the results ?

Mapping declared impact to funding received is one of the most interesting questions in RetroPGF, IMHO it’s the magic formula, since it’s a challenge to translate the variety of metrics declared to a coherent RetroPGF funding result, perceived as fair to the Collective and projects who applied.
See discussion on Quantifying Every Projects Impact as an OP Amount
@griff or The Role of VC Funding in RetroPGF

More thoughts (summarizing mostly what have been discussed in other RetroPGF3 feedback threads already):

  • running this analysis is relevant to understand past rounds, and could reveal super valuable insights for designing future rounds
  • Can/should declared impact metrics be verifyable and/or verified in the voting process? How?
  • Metrics standardization? Can and should it be a goal to standardize impact metrics that count for RetroPGF, such as VC Funding received or Sequencer Revenue Created @alexcutlerdoteth
  • Category Impact Metrics? Develop and define meaningful impact metrics per category, like number of active users in category End User UX, or #stars on Github in Developer Ecosystem to enable data-driven project comparison - which would be particularly valuable in case projects should be ranked in round 4.

Let’s explore these questions with GPT support!

1 Like

@latruite.eth @joanbp and everyone:
just updated our original post and added a link to book an onboarding call to RetroPGF GPT - just go ahead and reserve a slot, see you there!

2 Likes

Yop, would be awesome to get access to the impact data declared by projects in their applications. (Gotta be careful with the data integrity, of course.)
I’m fairly new to data analysis, but eager to explore I’m planning to do an article on a specific subgroup of projects: the .End User Experience Adoption: Evangelism & User Onboarding category (113 projects/individual applications).
Already booked a timeslot for an onboarding call , thx!

4 Likes

Very cool! Just booked a time to speak with you on 26 Feb

4 Likes