The bleu team is thrilled to be selected to execute the mission of building an AI-powered assistant designed to generate guidance and digested information about the Optimism Collective, more details here.
As outlined in our proposal, we’re creating this forum post to keep you updated on our progress and seek your valuable input every two weeks by posting a comment on this thread. Your feedback is crucial for us!
bleu is committed to transparency and open-source development. If you’d like to track our progress, feel free to check out our work on this GitHub Repository.
3 Likes
bleu Update #1
Milestones Tracker
Data Understanding (in progress)
Model Development (in progress)
Reporting Tool (to-do)
Assistant AI Launch (to-do)
Post-Launch Support (to-do)
Latest Activities
Reference Documents
We compiled a list of references, including documents crucial for training our model, primarily focusing on governance-related materials. The list of references is:
Data Collection
We developed Python scripts to collect data from governance documents and the forum. For the first model, we are focusing on these two sources, but we plan to collect data from additional sources in future iterations.
Q&A Database
Using language models, we generated a collection of questions and answers based on the data collected previously. This Q&A database will be used for evaluating our model’s performance.
Initial Model
We are developing our first AI chatbot model using the OpenAI API. This model will generate answers and utilize Retrieval-Augmented Generation (RAG) methods to effectively consult the reference documentation.
How You Can Help
Review the Document Reference List
Check if all the documents listed above are relevant and suggest any missing ones. As mentioned before, we’re mainly looking for documents related to the Optimism Governance.
Validate the Q&A Database
Ensure each question is relevant and the answers are accurate. You can access the Q&A database here. If you want to help us, please make a copy of the document, fill it out with your feedback, and share it here in the forum. If an answer is incorrect, please provide the correct one. Any additional feedback on the questions and answers, such as missing topics, is welcome.
2 Likes
Hi Bleu team! I’ve been answering some of the questions and leave the link here and keep doing it on my free time 81 from 231 question so far.
Link
2 Likes
bleu Update #2
Milestones
- Data Understanding (done)
- Model Development (in progress)
- Reporting Tool (in progress)
- Assistant AI Launch (upcoming)
- Post-Launch Support (to-do)
Latest Activities
Chat bot model development
We’ve been experimenting with various data pipelines and models for our chatbot. Each approach has been evaluated using our previously mentioned dataset. You can find a detailed breakdown of our experiments and results here: Project Overview - First Milestone.
UI Designs
We’ve created a design for the chatbot’s user interface, aiming for an intuitive user experience. You can view the design here: Chatbot UI Design.
For the reporting tool, we’re leaning towards using an existing product analyzer like Posthog. The post summaries UI is still in development and will be shared in our next update.
Project Architecture
To illustrate the overall structure of the project, we’ve created a comprehensive diagram. This outlines how our AI tools integrate into the final products: the Reporting Tool, Forum Summaries, and Chatbot.
Baseline Chatbot Launch
We’re excited to share that our baseline model is now available for community interaction. While it’s still a work in progress, you can access and test it here: Baseline Model Interface.
Form Summary Exploration
We’re in the early stages of developing models to summarize forum posts. Our current focus includes:
- Categorizing forum post types
- Developing pipelines for each post type
- Integrating Snapshot data where relevant
- Testing forum summaries as input for our main RAG model
Reporting tool
We have defined the key metrics to track across properties to be shown in the reporting tool. These metrics aim to help Optimism to analyze user behavior and gather valuable governance insights. Our preliminary list of metrics includes:
-
User Behavior
- Total Page Visits.
- Unique Page Visits.
- Average Session Duration.
- Tool Satisfaction (survey, randomly sampled across users).
-
Content Quality
- Content/Response Quality (thumbs down %).
-
Governance Insights:
- Most Searched Topics.
- Unanswered Questions.
How You Can Help
- UI Design Feedback: We welcome your thoughts on our current designs.
- Chatbot Testing: Your input here is crucial. Please test our chatbot here and use the
feedback button to report any issues!
- Reporting Tool Input: We’re open to suggestions on additional metrics or insights that would be valuable to track.
3 Likes
For the ones who will test the chatbot here is a feedback form as well: bleu's OP Govgpt Feedback Survey | Formbricks
3 Likes
bleu Update #3
Milestones
- Data Understanding (done)
- Model Development (in progress)
- Reporting Tool (in progress)
- Assistant AI Launch (upcoming)
- Post-Launch Support (to-do)
Latest Activities
Forum Summary Baseline Model
We’ve developed a model capable of summarizing forum posts with several key features. The model can categorize posts into various types, including:
- Discussion
- Feedback
- Announcement
- Guide
- Informative
- Unimportant
- Other
A different prompt strategy has been implemented for each category to ensure relevant summaries. Additionally, the model integrates current proposal state data from proposals to provide contextually rich summaries with a TLDR.
Chatbot Model Enhancement
After the baseline chatbot launch, we gathered user feedback through a UX survey to identify areas needing improvement. Based on this feedback, we have implemented several enhancements which should be available next week. These include:
- Data Enrichment: Utilizing forum summaries as additional input
- Query Expansion: Implementing query expander for a more comprehensive understanding
- Advanced Retrieval: Integrating iterative retriever techniques.
We are currently testing and evaluating these enhancements to ensure their effectiveness.
Forum Summary Application Development
We have initiated the integration of the baseline model into a server environment. Alongside, we have designed and implemented a user interface for accessing forum summaries, which is still WIP and we’ll share a link to it for feedback before the next update.
How You Can Help
- Keep testing the chatbot: Since we’re still improving the chatbot your feedback is highly valuable. You can test our chatbot here and use the feedback button to report any issues! Also, you can also answer our UX survey.
- Check the forum summaries: Test the current state of our forum summary application and provide feedback about the UI and the summaries. You can send your feedback in this thread or directly on Telegram @yvesfracari.
3 Likes
bleu Update #4
Hey folks! Coming up with another update here.
Milestones
- Data Understanding (done)
- Model Development (in progress)
- Reporting Tool (closing)
- Assistant AI Launch (upcoming)
- Post-Launch Support (to-do)
Latest Activities
Enhancing the chatbot model is proving more challenging than initially anticipated in the grant proposal. While we’ve exceeded the project budget already, delivering good results remains our priority. We are committed to continuing work on this project and addressing community feedback until we feel satisfied with what we’ve delivered.
Forum Summary Application Development
We’ve embedded the post summarizer within our app. It’s currently running on a forum snapshot, as real-time database updates are still in progress. View it here
Reporting Tool
We’ve also created product dashboards to track usage and identify improvement areas:
Chatbot Model Enhancement
We acknowledge our initial chat model fell short of expectations. However, this early release provided invaluable feedback for us to keep improving. Thank you for your continued support!
Since our last update, we’ve substantially refactored the chatbot model architecture. Here’s what we’ll be shipping in the next days:
- Added conversation memory
- Indexed common questions
- Enhanced time understanding
- Refined prompt strategies for different types of questions (factual/abstract/time-bound)
Thus here’s what you should expect (to be live soon!) from this iteration:
- More accurate and contextual responses to time-related queries
- Improved handling of frequently asked questions
- More reliable and precise answers to factual questions
How You Can Help
- Chatbot Testing
- Forum Summaries Feedback:
- Test and provide UI/summary feedback
- Use
and to react to summaries 
- Share feedback in this post or via Telegram @yvesfracari
- Reporting Tool Feedback:
- General feedback welcome!
4 Likes
bleu Update #5
Hey there! Coming up with another update:
Milestones
- Data Understanding (done)
- Model Development (closing)
- Reporting Tool (closing)
- Assistant AI Launch (upcoming)
- Post-Launch Support (to-do)
Latest Activities
As mentioned in our last update, our focus has been on refining the chatbot to ensure it delivers meaningful value to the community and aligns with the mission request objectives.
Specifically, we concentrated on:
- enhancing the chatbot’s sensitivity to time-related questions;
- addressing a context filtering error and improving the prompts;
- refactoring the codebase for better performance and maintainability;
- implementing infrastructure improvements for ease of maintenance - specifically updating forum summaries and data that the chat uses to respond to questions.
Chatbot Model Enhancement
Our efforts to improve the chatbot model have showed significant results. We’ve addressed several technical challenges and observed consistent improvements in the model’s performance.
- Time-sensitive questions: The model now more accurately distinguishes between historical and current data.
- Filtering error: Resolved a critical issue in the context filtering mechanism, leading to more accurate and contextually appropriate responses.
- More effective prompts: Refined prompts result in higher quality outputs across various query types.
- Infrastructure improvements: Refactored database connections and implemented OpenAI’s structured output API to allow for more consistent response formats from the LLM.
Additionally, we performed a light refactoring of the op-brains system, which has contributed to the overall improvement in performance. Our internal testing, including multiple runs of specific queries, has shown consistent and accurate responses.
Next Steps
- Push for wider community testing of the chatbot
- Continue documentation efforts for better maintainability
- Explore improvements in summarization of important topics
We’re optimistic about these enhancements and look forward to gathering community feedback on this latest version!
How You Can Help
3 Likes
Data Understanding (done)