Hello GC and thank you @Gonna.eth for putting this proposal together! As part of my role as a member of the Collective Feedback Commission, I’ve been asked to provide feedback on this Operating Budget proposal.
I will focus on the KPI’s for the Season:
1. Increase in scope for NPS
Appreciate that the Council is now moving to evaluate the NPS not only for grantees but also to applicants. This will give us more information on how to improve all builders experience through the Grant process.
I would suggest two changes: i) include an exit survey after a proposal is submitted, and another survey after a final outcome has been published for each proposal.
The goal here is to measure at least two points in the user journey of an applicant, by doing this the GC will have better information for improvements at each stage.
- Clarity and ease of the Submission Process (ask from Charmverse for 3-5 short questions to be triggered upon clicking on submit or add it as a final step in the submission).
- Quality of interactions with the GC during the grant review process and clarity of the outcome (include this survey as an email or in the forum post with the results).
ii)New split in NPS reporting based on the above:
- 7/10 for all applicants at the submission stage
- 7/10 for rejected applicants at the final decision stage
- 9/10 for accepted applicants at the final decision stage
This split in data reporting will enable us to keep pushing for an outstanding score at the grantee stage (as achieved in previous seasons) and slowly ramp up for a similar results with applicants.
This breakdown shouldn’t represent an increase in overhead since the information can easily be split based on applicant status.
It is important to note that there is evidence in user satisfaction evaluation that for the final stage there will be timing bias and sample bias. TLDR; Accepted grantees scoring the GC after acceptance and in the questionnaire where they input disbursement info will naturally rate their experience higher. This is not bad, but we should be aware of it.
2. More Analysis on User Experience
I strongly agree with the feedback provided to the GC Season 6 Retrospective around expanding how we analyze and understand builder satisfaction;
I would suggest the GC to pilot including 2-4 open ended questions in each of the survey points mentioned above and conduct an AI-led Descriptive Discourse Analysis on the answers to identify:
- Common themes (pain points or outstanding interactions in the process)
- Applicant Sentiments (frustration, satisfaction, etc).
The analysis of this information could be conducted by the GrantNerds, the M&M Council, or someone else (SeedLatam in their Season Report?), but data gathering is best conducted by the GC as the owner of the process.
The KPI at this initial stage can be to have a response rate of at least 70% of all applicants to both surveys. Once we have more data, the GC can determine more granular KPIs.
The GC is part of the public perception of Optimism and the ease and support builders receive in the ecosystem, implementing better ways to measure and improve the quality of interactions builders have can drive better builder retention.
3. TVL projections
In addition to the Baseline Goal of TVL growth for all grants, I would suggest including the other component of this analysis as KPI.
As in:
| At Program End Date (Or Latest) based on n | Incremental TVL per Year / OP |
|---|---|
| 75th Percentile | $37.93 USD |
| 50th Percentile (Median) | $13.83 USD |
| 25th Percentile | $0.73 USD |
*where n = all grants disbursed in the Season
This will enable a better evaluation per grant approved rather than an overall amount which could be skewed by just one outlier project (in this case both 25th and 75th percentile would be very very low).
Questions:
- I’m not sure I follow what you mean by “TVL metrics driven by approved grants” as part of the Cycle Reports. Could you please expand on it? I think it might be a breakdown of how many or which grants support each of these, but I’m not sure.
I hope this feedback is useful to strengthen the tools that the Grants Council has at its disposal to measure its efficacy and contributions to Seasons 7 Intent.