The bleu team is thrilled to be selected to execute the mission of building an AI-powered assistant designed to generate guidance and digested information about the Optimism Collective, more details here.
As outlined in our proposal, we’re creating this forum post to keep you updated on our progress and seek your valuable input every two weeks by posting a comment on this thread. Your feedback is crucial for us!
bleu is committed to transparency and open-source development. If you’d like to track our progress, feel free to check out our work on this GitHub Repository.
We compiled a list of references, including documents crucial for training our model, primarily focusing on governance-related materials. The list of references is:
We developed Python scripts to collect data from governance documents and the forum. For the first model, we are focusing on these two sources, but we plan to collect data from additional sources in future iterations.
Q&A Database
Using language models, we generated a collection of questions and answers based on the data collected previously. This Q&A database will be used for evaluating our model’s performance.
Initial Model
We are developing our first AI chatbot model using the OpenAI API. This model will generate answers and utilize Retrieval-Augmented Generation (RAG) methods to effectively consult the reference documentation.
How You Can Help
Review the Document Reference List
Check if all the documents listed above are relevant and suggest any missing ones. As mentioned before, we’re mainly looking for documents related to the Optimism Governance.
Validate the Q&A Database
Ensure each question is relevant and the answers are accurate. You can access the Q&A database here. If you want to help us, please make a copy of the document, fill it out with your feedback, and share it here in the forum. If an answer is incorrect, please provide the correct one. Any additional feedback on the questions and answers, such as missing topics, is welcome.
We’ve been experimenting with various data pipelines and models for our chatbot. Each approach has been evaluated using our previously mentioned dataset. You can find a detailed breakdown of our experiments and results here: Project Overview - First Milestone.
UI Designs
We’ve created a design for the chatbot’s user interface, aiming for an intuitive user experience. You can view the design here: Chatbot UI Design.
For the reporting tool, we’re leaning towards using an existing product analyzer like Posthog. The post summaries UI is still in development and will be shared in our next update.
Project Architecture
To illustrate the overall structure of the project, we’ve created a comprehensive diagram. This outlines how our AI tools integrate into the final products: the Reporting Tool, Forum Summaries, and Chatbot.
Baseline Chatbot Launch
We’re excited to share that our baseline model is now available for community interaction. While it’s still a work in progress, you can access and test it here: Baseline Model Interface.
Form Summary Exploration
We’re in the early stages of developing models to summarize forum posts. Our current focus includes:
Categorizing forum post types
Developing pipelines for each post type
Integrating Snapshot data where relevant
Testing forum summaries as input for our main RAG model
Reporting tool
We have defined the key metrics to track across properties to be shown in the reporting tool. These metrics aim to help Optimism to analyze user behavior and gather valuable governance insights. Our preliminary list of metrics includes:
User Behavior
Total Page Visits.
Unique Page Visits.
Average Session Duration.
Tool Satisfaction (survey, randomly sampled across users).
Content Quality
Content/Response Quality (thumbs down %).
Governance Insights:
Most Searched Topics.
Unanswered Questions.
How You Can Help
UI Design Feedback: We welcome your thoughts on our current designs.
Chatbot Testing: Your input here is crucial. Please test our chatbot here and use the feedback button to report any issues!
Reporting Tool Input: We’re open to suggestions on additional metrics or insights that would be valuable to track.
We’ve developed a model capable of summarizing forum posts with several key features. The model can categorize posts into various types, including:
Discussion
Feedback
Announcement
Guide
Informative
Unimportant
Other
A different prompt strategy has been implemented for each category to ensure relevant summaries. Additionally, the model integrates current proposal state data from proposals to provide contextually rich summaries with a TLDR.
Chatbot Model Enhancement
After the baseline chatbot launch, we gathered user feedback through a UX survey to identify areas needing improvement. Based on this feedback, we have implemented several enhancements which should be available next week. These include:
Data Enrichment: Utilizing forum summaries as additional input
Query Expansion: Implementing query expander for a more comprehensive understanding
We are currently testing and evaluating these enhancements to ensure their effectiveness.
Forum Summary Application Development
We have initiated the integration of the baseline model into a server environment. Alongside, we have designed and implemented a user interface for accessing forum summaries, which is still WIP and we’ll share a link to it for feedback before the next update.
How You Can Help
Keep testing the chatbot: Since we’re still improving the chatbot your feedback is highly valuable. You can test our chatbot here and use the feedback button to report any issues! Also, you can also answer our UX survey.
Check the forum summaries: Test the current state of our forum summary application and provide feedback about the UI and the summaries. You can send your feedback in this thread or directly on Telegram @yvesfracari.
Enhancing the chatbot model is proving more challenging than initially anticipated in the grant proposal. While we’ve exceeded the project budget already, delivering good results remains our priority. We are committed to continuing work on this project and addressing community feedback until we feel satisfied with what we’ve delivered.
Forum Summary Application Development
We’ve embedded the post summarizer within our app. It’s currently running on a forum snapshot, as real-time database updates are still in progress. View it here
Reporting Tool
We’ve also created product dashboards to track usage and identify improvement areas:
We acknowledge our initial chat model fell short of expectations. However, this early release provided invaluable feedback for us to keep improving. Thank you for your continued support!
Since our last update, we’ve substantially refactored the chatbot model architecture. Here’s what we’ll be shipping in the next days:
Added conversation memory
Indexed common questions
Enhanced time understanding
Refined prompt strategies for different types of questions (factual/abstract/time-bound)
Thus here’s what you should expect (to be live soon!) from this iteration:
More accurate and contextual responses to time-related queries
Improved handling of frequently asked questions
More reliable and precise answers to factual questions
As mentioned in our last update, our focus has been on refining the chatbot to ensure it delivers meaningful value to the community and aligns with the mission request objectives.
Specifically, we concentrated on:
enhancing the chatbot’s sensitivity to time-related questions;
addressing a context filtering error and improving the prompts;
refactoring the codebase for better performance and maintainability;
implementing infrastructure improvements for ease of maintenance - specifically updating forum summaries and data that the chat uses to respond to questions.
Chatbot Model Enhancement
Our efforts to improve the chatbot model have showed significant results. We’ve addressed several technical challenges and observed consistent improvements in the model’s performance.
Time-sensitive questions: The model now more accurately distinguishes between historical and current data.
Filtering error: Resolved a critical issue in the context filtering mechanism, leading to more accurate and contextually appropriate responses.
More effective prompts: Refined prompts result in higher quality outputs across various query types.
Infrastructure improvements: Refactored database connections and implemented OpenAI’s structured output API to allow for more consistent response formats from the LLM.
Additionally, we performed a light refactoring of the op-brains system, which has contributed to the overall improvement in performance. Our internal testing, including multiple runs of specific queries, has shown consistent and accurate responses.
Next Steps
Push for wider community testing of the chatbot
Continue documentation efforts for better maintainability
Explore improvements in summarization of important topics
We’re optimistic about these enhancements and look forward to gathering community feedback on this latest version!
Hey everyone! We’re excited to share our final update on the GovGPT and GovSummaries project. We’ve reached the completion of our mission to build an AI-powered assistant for the Optimism Collective.
Milestones Tracker
Data Understanding (done) Model Development (done) Reporting Tool (done) Assistant AI Launch (done) Post-Launch Support (ongoing)
Latest Activities
Chatbot Model Finalization
Completed refactoring of the chatbot model architecture
Implemented conversation memory for context-aware responses
Enhanced time understanding for accurate historical and current data distinction
Refined prompt strategies for various question types (factual/abstract/time-bound)
Forum Summary Application
Fully integrated the post summarizer within the app
Implemented real-time database updates for forum summaries
Finalized UI for accessing forum summaries
Reporting Tool Completion
Finalized product dashboards for tracking usage and identifying improvement areas:
Feedback: Use the to report any issues or inaccuracies
Reporting Tool: Any suggestions for additional metrics or insights are welcome!
We want to express our gratitude to the Optimism community for your support and feedback. While we’ve officially completed the project milestones, our commitment to improving and maintaining GovGPT continues. We’re excited to see how this tool will enhance engagement within the Optimism ecosystem! Thank you!