[REJECTED] Real-time access to pre-chain data RFP Mission Request

Governance Fund Missions 🏹
,
1

Delegate Mission Request Summary:

Improving observability of the pre-chain layers of Optimism based superchains so that developers have equitable access to the necessary data. This will enable developers to optimize their offerings, enhance UX, address stuck wallets, and build new experiences – improving trust and understanding around the settlement and aiding the superchains in attracting developers, users, institutional investors, and more.

S5 Intent: Intent 1: Progress towards technical decentralization

Proposing Delegate: Gonna.eth (Sponsor)

Proposal Tier: Eagle Tier

Baseline grant amount: 650k OP

Should this Foundation Mission be fulfilled by one or multiple applicants: Multiple

Submit by: To be set by Grants Council

Selection by: To be set by Grants Council

Start date: If applicable

Completion date: 12 months upon grant approval

Specification

How will this Delegate Mission Request help accomplish the above Intent?

In order to ensure Optimism is a level playing field, all actors need to be able to verify the orderly settlement of the chain. Today users and developers do not have access to the pre-chain layer (Sequencer).

In order to put power in the hands of people everyone needs access to the same information. Enabling access to the Sequencer via credibly neutral entities will remove centralizing incentives and provide the critical component for protocol transparency and data accessibility.

To onboard the next wave of blockchain users, developers need comprehensive observability into the chain’s inner workings to create best in class user experiences. The interoperability between Ethereum and the Optimism network relies heavily on Optimism’s canonical bridge contracts. Observability into these is important for many reasons:

  • [Foundation/core devs] Security Monitoring: Real-time monitoring of contract activity to identify and address security vulnerabilities promptly.
  • [Foundation/core devs] Network Health Analysis: Enables continuous assessment of network performance, guiding improvements and upgrades.
  • [Builders] Developer Insight: Provides developers with critical data for optimizing smart contract design and interaction.
  • [Builders] Troubleshooting Efficiency: Quick diagnosis and resolution of issues in asset transfers, enhancing operational reliability.
  • [Web3 users/projects] User Trust: Transparent transaction tracking enhances user confidence in the integrity of the asset transfer process.
  • [Web3 users/projects] Compliance and Reporting: Facilitates regulatory compliance with detailed transaction histories and contract interaction logs.

Real-time access to pre-chain data will facilitate:

  • Composability for developers
  • Rich API access for developers to programmatically consume data
  • Backward looking analysis to monitor in real-time composed metrics like liquidity deposited on a protocol
  • A robust data set for protocol developers and researchers to improve the functioning of the network and their application
  • Real-time notifications for application and wallet developers and their end-users
  • Stuck transaction detection for application and wallet developers and their end-users
  • Gas estimation tooling for posting batches
  • Real-time notifications for all bridge transactions L1<> L2 and L2<>L1 giving full observability between Optimism and the base layer

This mission will contribute to a more observant and equitable ecosystem which, at its basis, is critical to bringing power to the people.

What is required to execute this Delegate Mission Request?

  • Access to the Sequencer
  • Standing up and running Optimism archive nodes
  • Storage capability to log pre-chain & on-chain data
  • Front-end design UI for open source explorer
  • API support capabilities

How should the Token House measure progress towards this Mission?

  • Research: Receiving access to Sequencer and researching/understanding behavior [2 months from grant date]
  • Archive Development: Launch Sequencer data database and UX [4 months from grant date]
  • Real-time APIs: Developer API access to pending transactions, security monitoring, batch transactions, bridge transactions, etc. [6 months from grant date]
  • Data Analytics Platform: Front-end, open source UX for equal access to pre-chain data [9 months from grant date]
  • Gas Estimation Tooling: L2 gas estimation based on current pre-chain activity instead of past blocks [12 months from grant date]

How should badgeholders measure impact upon completion of this Mission?

  • Developers have programmatic access to the necessary pre-chain data
  • User facing resources to observe and check transaction status
  • Utilities for diagnosing stuck wallets and other pain points
  • Improved gas estimation on Optimism

Have you engaged a Grant-as-a-service provider for this Mission Request?
No.

Has anyone other than the Proposing Delegate contributed to this Mission Request?

Consulted the Blocknative team and Jack Anorak.

4 Likes
2

I am an Optimism delegate with sufficient voting power and I believe this proposal is ready to move to a vote.

3 Likes
3

I am an Optimism delegate with sufficient voting power and I believe this proposal is ready to move to a vote.

2 Likes
4

I am a delegate with sufficient voting power and I believe this proposal is ready to move to a vote.

2 Likes
5

While it feels very ambitious, you’re one short and I think this could be an interesting use of resources, so supporting it, best of luck!

I would like to see this pushed to the next stage, so I am an Optimism Delegate with sufficient voting power and I believe this proposal is ready to move to a vote.

3 Likes
6

This is an interesting one. One thing that piques my interest is:

Access to the Sequencer

Currently its centralized under Optimism control and giving access to the private mempool gives privileged access.

edit: I’m also an employee of OP Labs and speaking on my own behalf.

3 Likes
7

I am an employee of OP Labs and speaking on my own behalf.

This post describes a really complicated way to say “public mempool” or “preconfirmations”. The sequencer already provides preconfirmations aka “unsafe blocks” that are sent via a p2p network to give low latency access to the data that will eventually be finalized (settled to L1).

What does “verify the orderly settlement of the chain” mean? Orderly as in fast? Non equivocating? Or something else? A much more clear description is required if you expect other people to have a shared understanding of what you mean here.

This is very not clear what you mean. If you mean unsafe blocks, this is just not true. I mentioned above that the sequencer sends preconfirmed blocks via p2p which gives good UX to end users. In practice this looks like transactions instantly being visible on Etherscan. Preconfirmed blocks have not been published to the DA layer yet (L1) but will be in the future. If you mean mempool, you should consider the second order effects of a public mempool (ie toxic MEV extraction like sandwiching). I am personally in favor of a public mempool eventually although private mempools are likely the future due to the economies of scale with blockchain scalability.

Reading through your post, it is very unclear what you are asking for. What needs to be observed? There are no quantitative examples, everything is very handwavy and qualitative. Developers don’t need access to the mempool to optimize their smart contracts. The required things to execute the mission request are a bunch of random words put together. There is no cohesive story describing what the problem and solution are.

  • Stuck transaction detection for application and wallet developers and their end-users

This is actually a real problem to be solved! Yay! But unclear how the proposal has anything to do with solving it. In practice it could be solved with some sort of PBS.

5 Likes
8

my question to Blocknative: are you the only organization currently working on something like what you’re proposing?

2 Likes
9

I am an employee of OP Labs; My views are my own.

First, I agree with @tynes, Access to pre-chain data is already available via the unsafe-head which is gossiped. Anyone who wants to provide “Real-time access to pre-chain data” can do exactly that as soon as the block has been gossiped to them. Any Web2 SAAS can receive the unsafe head via gossip and can serve information about it out to its customers. Doing so would indeed be faster than the on-chain data, because it is not the safe head.

I also want to add that L1 Deposited transactions serve to provide Data Accessibility, entirely unrelated from the Sequencer. If users would like to ensure their transactions are settled in an orderly way, submitting their transactions to L1 accomplishes that.

However, this proposal seems to be focused on mempool access specifically, so I wanted to respond to the idea of having programatic access to the L2 mempool: Optimism uses a centralized sequencer, meaning there is only one ‘correct’ view of the mempool, and it is located on that node. Because of this, the sequencer would be directly responsible for serving this information.

If the sequencer were to respond to API calls to fetch its mempool data, we would have to decide how broadly to allow that access. For truly equitable access, this API would need to be free and publicly accessible. This is not technically feasible without a considerable amount of Web2 infrastructure to handle the load. Instead, this proposal references “credibly neutral entities” having this API access only, presumably owning the infrastructure related to this effort. This is less-than equitable, as mempool-as-a-service tends to be a paid Web2 service. It also opens the question of what it means to be a “credibly neutral entity”.

Any amount of interference or additional load into the sequencer brings the risk of performance tradeoffs. Currently, the sequencer is responsible for building and gossiping the unsafe-head. Interfering with the Sequencer in order to export that data faster seems to benefit only those who are approved to consume the data.

This would would also open Optimism up to new types of MEV which have not previously been a concern. A transaction submitted to the L2 has the benefit of transaction privacy until their Tx is published in an unsafe block, at which point MEV is not possible. By contrast, if a service were to export the sequencer’s mempool to its customers ahead of unsafe-blocks, those customers would have the opportunity to front-run transactions in the block. Actors using this API would be given a competitive advantage in the dark forest, creating inequity.

From my perspective, this proposal is seeking something which already exists (pre-chain data), delivered to a subset of actors (credibly neutral entities) for the purposes of building new Web2 services akin to the Mempool Explorer Service that Blocknative already sells to customers. I do not think it enhances equability, as this proposal only demonstrates privileged access to pre-chain data.

Looking at the features listed as being enabled by this faster pre-chain access, I don’t understand how the proposal achieves them:

  • “Composability”, I’m not sure what composability is unlocked here, seeing as all this is happening off-chain.
  • “Rich API” is a tautology, as that’s the essence of what’s being proposed here
  • “Backward-looking Analysis” and “Robust Data Set” are achievable regardless of data speed.
  • “Real-time notifications (2x)” and “Gas Estimation” are achievable by the unsafe head, and if a company would like to serve data from it to accelerate their customers, they may do so.
  • “Stuck transaction detection” would be nice, but like tynes said, it’s not clear how this proposal achieves this besides interpreting the real-time data (which, as above, could be done against the unsafe-head)
6 Likes
10

Hello @tynes and @AxelKingsley, I want to express my sincere gratitude for your valuable time and engagement. Allow me to provide further context.

I’ve been actively supporting numerous mission requests on behalf of teams and individuals contributing to the optimism-driven initiatives. My consistent conditions for sponsorship have been centered around ensuring that mission requests are open for fair competition among applicants and that the final product remains open source.

Upon reviewing your insightful comments, I acknowledge that this specific mission request exceeds my general knowledge. I appreciate your feedback and enlightenment on the matter. As a mere sponsor of this proposal, I’ve taken the initiative to involve experts from Blocknative to engage in a robust and intellectually stimulating debate—a discourse that I personally find both enriching and enjoyable.

It’s essential for me to emphasize that I will reconsider my sponsorship if the mission request attracts only a single applicant and is geared towards creating a monetized service rather than an open-source, freely accessible tool for fellow developers. My commitment lies in supporting initiatives that promote inclusivity, fairness, and the broader benefit of the developer community.

Thank you both for helping me in this endeavor and for your invaluable assistance in navigating these considerations. Your expertise is truly appreciated.

8 Likes
11

We thank everyone for providing thoughtful and comprehensive feedback on our Mission Request. Overall, your feedback has forced us to think more critically about the intent, structure, and format of the Mission Request.

Based on the feedback received, we are now revising/revamping the Mission Request to more clearly articulate the core problems to be solved (various classes of real-time observability, gas estimation/prediction, stuck transactions) and recommended phases for addressing each problem (for instance, starting by providing end-to-end transaction status during L2<>L1 finalization). Our aim is to make the Mission Request clearer, stronger, and disambiguated. We hope to share it here in the next several days.

For the avoidance of doubt, this revised Mission Request will make it clear that:

  • We are not proposing any one party or parties be granted privileged access at any point in the stack.
  • The process of decentralizing infrastructure creates the conditions for information asymmetries to emerge – particularly within the real-time ephemeral data layers that affect/determine settlement.
  • The ecosystem has the opportunity to fund the development and operation of public goods that reduce/eliminate these information asymmetries, level the playing field for all participants, and encourage/support further research and understanding in these areas.
  • The Mission Request will propose a grounded, phased approach that moves in step with the policies of the underlying infrastructure.
  • Multiple teams are actively building tooling and infrastructure that involves these ephemeral data layers. Many may be interested in learning more and applying for grants under this Mission Request.

Our revised Mission Request will directly address most/all of the questions raised by tynes, jackanorak, and AxelKingsley.

5 Likes
12

We have submitted our revised Mission Request for internal review and we look forward to sharing it here soon.

1 Like
13

Thank you all for the rich discussion and feedback. Here is our revised Mission Request based on what we have heard from the community. We look forward to continuing the discussion:

Public Goods Infrastructure to Enhance Observability in Service of Transparency, Developer Experience, and Decentralization

S5 Intent : Intent 1: Progress towards technical decentralization
Proposing Delegate: Gonna.eth (Sponsor)
Proposal Tier: Eagle Tier
Baseline grant amount: 650k OP
Should this Foundation Mission be fulfilled by one or multiple applicants: Multiple
Submit by: To be set by Grants Council
Selection by: To be set by Grants Council
Start date: If applicable
Completion date: 12 months upon grant approval

Delegate Mission Request Summary:

This Mission Request aims to enhance the observability of the Optimism ecosystem by incentivizing the development of public goods tooling and infrastructure. This Mission Request demonstrates how improvements in observability can directly translate into real-world solutions to pressing challenges, such as clarifying the end-to-end transaction lifecycle, mitigating stuck transactions, improving gas estimation and gas prediction (including the complexities introduced by EIP4844), or enabling archival access to ephemeral datasets including type 3 blob transactions.

Specification

How will this Delegate Mission Request help accomplish the above Intent?

This Mission Request focuses on improving the observability of the Optimism ecosystem via new public goods tooling and infrastructure. As the Optimism ecosystem makes progress on its decentralization roadmap, it is desirable to ensure that all ecosystem participants have equitable access to the data layers that relate to transaction settlement. These layers are becoming increasingly complex as the base layer evolves (new EIPs) and as the Optimism ecosystem evolves into the Superchain, making it critical to address the observability challenges today and research approaches for the future.

The intent of this Mission Request is to encourage researchers, builders, and infrastructure providers to submit grant proposals which yield positive progress towards implementing sustainable long-term solutions to these observability challenges.

Real-time observability covers, but is not limited to, the following problems:

  • End-to-End Transaction Status & the Stuck Transaction Problem
    • The end-to-end L2 transaction lifecycle has many stages that developers and end-users need to know:
      • Nominal Lifecycle: Pending L2 > Sequencer Confirmed / Unsafe Block L2 > Batch Pending L1 > Published to Ethereum / Safe L1 > Finalized L1
      • Other Possible Statuses: Underpriced L2, Stuck L2, Dropped L2, Batch Underpriced L1, Batch Stuck L2, Batch Dropped L2, Batch Reorged L2
      • With the introduction of EIP4844, there may be additional gaps in the L2 transaction lifecycle, including: pending L1 normal (type 0 or type 2) transaction vs. pending L1 blob (type 3) transaction
      • And potentially additional transaction status types enabled by other upcoming Ethereum roadmap milestones
    • While some transaction status types are available, others are more difficult to ascertain or unavailable.
    • Some examples of relevant tooling and infrastructure:
      • Transaction status APIs for wallets and traders
      • Smart RPCs (for example, ones that take into account gas prices and/or user preferences)
      • Visual interfaces for transaction’s status through L1 finality
  • Gas Estimation & Prediction
    • With the increasing complexity of the transaction lifecycle via the introduction of EIP4844, there is an opportunity to incentivize a more robust and diverse landscape for real-time gas estimation. For example, an L2 transaction will have to account for EIP1559 on the L2, EIP1559 on the L1 for regular transactions, or EIP1559 for blob transactions depending on how batches are settled to the L1 – also referred to as multi-dimensional gas. There will be times when regular type 2 batch transactions are more expensive than type 3 blob transactions and vice versa. So, incentivizing the development and operation of a diverse set of gas estimation tooling will improve developer and user experience while saving end-users and protocol providers money on transaction fees.
    • As recently demonstrated by the inscription craze, network congestion can cause L2 batches to become stuck in the Ethereum public mempool, which in turn can negatively impact the L2 developer and user experience. The introduction of EIP4844 may increase both the occurrence and complexity of these situations. Net new public goods tooling, APIs, and observability will be needed to monitor both the L1 mempool and the blob mempool.
    • Comprehensive gas estimation at each layer of the stack will help reduce stuck transactions, whether at the L2 or L1.
    • Some examples of relevant tooling and infrastructure:
      • Real-time machine learning based Optimism gas prediction via API and/or visual interfaces
      • 4844 blob pricing prediction modeling
  • Archival Access to Ephemeral Datasets
    • EIP4844 introduces type 3 transactions (blobs) mainly geared towards L2 rollups as a cheaper alternative for posting their L2 batch transactions to Ethereum.
    • The actual blob data will only be accessible for 4,096 epochs (~ 18 days) on the beacon nodes (CL).
    • For the Optimism community to have historical data access for research purposes and integration with block explorers, multiple parties should be incentivized to fetch, store, and open source the blob data.
    • Some examples of relevant tooling and infrastructure:
      • The collection and maintenance of these data sets for the community - accessible as a public good via APIs
      • Visual interfaces such as dashboards, explorers, etc.

Grants issued under this Mission Request will be for the design and development of real-world production solutions as well as to fund the necessary operational infrastructure to supply the public good on an ongoing basis.

What is required to execute this Delegate Mission Request?

  • The teams participating in this Mission should have members skilled in implementing the topics referenced above.
  • Developers need extensive experience in the relevant development field. Having a history of contributing to related open-source projects is a plus.
  • Teams should have experience running infrastructure at scale in Web3.
  • The Mission aims at practical applications that enhance the usability of the Superchain. This requires a combination of theoretical understanding and hands-on technical expertise with Web3 infrastructure.
  • Deliverables are open-sourced to the public and provided as a public good.
  • It’s preferable for one entity to submit a single proposal covering multiple topics, rather than multiple proposals unnecessarily.
  • To avoid any doubt, grants issued will explicitly prohibit any form of proprietary and/or exclusive access to any data layer or data set.

How should the Token House measure progress towards this Mission?

  • Publicly accessible real-time transaction explorers for developers and users to observe transaction status.
  • Programmatic access via real-time developer APIs.
  • Improved gas estimation on Optimism.
  • Utilities for detecting, diagnosing, and repairing stuck wallets and other pain points.

How should badgeholders measure impact upon completion of this Mission?

  • Assess the extent to which the tools are being utilized and adopted within the community and target users.
  • Github key metrics such as Issues, Pull Requests, Stars, Watches, and Forks.

Have you engaged a Grant-as-a-service provider for this Mission Request?
No.

Has anyone other than the Proposing Delegate contributed to this Mission Request?
Blocknative team.

3 Likes
14

Two things.

  1. I don’t see exactly how the new mission request addresses the concerns of @AxelKingsley and @tynes. I don’t see this as technically feasible at this point of the maturity of optimism’s decentralization map. And having employees of OPlabs chime in and say so here probably should be enough to clarify that this is not a good idea.

  2. As a mission request it really seems as if something tailored to be completed directly by blocknative as @jackanorak also said. Which is against the intention of mission requests. A mission request is intended to be an RFP for multiple entities to apply to solve. Mission requests made to be fullfilled by a single entity were withdrawn. An example here: [WITHDRAWN] RetroPGF for any DAO mission request

1 Like
15

Thank you for weighing in here @lefterisjp .

In our revision, we updated the language to further highlight the open collaborative nature of this Mission Request. Some of the organizations well positioned to apply for grants under this Mission Request include (but are not limited to) GasHawk, Viem, Owlracle, Blockscout, Etherscan, Dune, and The Graph. In the interest of clarity, the revised Mission Request includes specific examples of how the concepts could come to life as well as requiring that all grant recipients deliver open source public good solutions.

To address the different points further:

We understand that third-party access (by Blocknative or anyone else) to the sequencer is not feasible at this time – hence the revised Mission Request does not include any mention of sequencer access. To the contrary: “To avoid any doubt, grants issued will explicitly prohibit any form of proprietary and/or exclusive access to any data layer or data set.”

&

Again, we agree that opening sequencer access is not part of the short-term roadmap for community.

However, unsafe blocks and pending transactions are not equivalent in terms of observability. For example: an unsafe block state does not show how much time there was between when a transaction entered the sequencer mempool and when it was confirmed into an L2 unsafe block. It also does not show how the sequencer ordered the transactions in the unsafe block.

The revised Mission Request removes any mentions of ‘orderly settlement’ due to the feedback above. In the interests of clarity – and reducing hand-waviness – the revised Mission Request specifies each stage of the end-to-end transaction lifecycle as described below (from the revised Mission Request):

We debated if this was too much detail but ultimately chose to provide more, rather than less, detail.

&

Again, the revised Mission Request eliminates any consideration of exposing the sequencer mempool.

With that said, our team has provided infrastructure and developer tooling in the MEV space for multiple years. We currently operate free public MEV protection endpoints. Based on our experience, we are interested in participating in discussions around the implications, impact, and tradeoffs that different approaches might have on the Optimism ecosystem. Is there a public working group focused on these topics?

The revised Mission Request includes details specific to gas estimation and prediction, stuck transactions, 4844/blobs, archival data capture/storage/access, and more.

Again, we agree.

&

We understand. Internally, we discussed how delayed data streams may be a productive and practical compromise to enabling research in these areas. The intent would be to open new avenues of research without providing privileged low-latency access.

Ultimately, we did not not include this line of thinking in the revised Mission Request. However, if the community is interested, this might be worth further exploration.

As stated above, unsafe blocks and pending transactions are not equivalent.

Again, the revised Mission Request states “grants issued will explicitly prohibit any form of proprietary and/or exclusive access to any data layer or data set”.

Blocknative’s three most recent offerings are each public goods:

Together these provide enhanced observability, enable new academic research pathways, and provide programmable MEV protection to the broader Ethereum ecosystem. Further, we have operated and maintained the widely-deployed open source Web3 Onboard library for 5+ years which includes capabilities specific to the Optimism ecosystem.

I hope this concern has been thoroughly addressed throughout the comments above as well as explicitly in our revised Mission Request.

We deeply appreciate the constructive engagement during this process as we navigate our first Mission Request.

We are excited to continue supporting the Optimism ecosystem and are always looking for the best way to do so. We welcome all suggestions from Delegates on areas where collaboration would be most impactful.

2 Likes
16

I am an Optimism delegate with sufficient voting power and I believe this proposal is ready to move to a vote.

17

Personally I don’t know enough about this at the moment to make an informed decision of whether or not it is worth 650k OP. The grant amount seems excessive… but maybe it will be worthwhile. Either way, there are enough approvals so mine is not needed.

18

The Developer Advisory Board has reviewed this Delegate Mission Request, and voted on its acceptance or rejection. The vote results are as follows:

ACCEPT: 0 votes
REJECT: 6 votes
ABSTAIN: 0 votes

therefore, the Developer Advisory Board rejects this delegate mission request.

The Developer Advisory Board has reached this conclusion for a number of reasons:

  1. It does not move the Optimism ecosystem towards technical decentralization. In our view, this proposal could be a centralizing force until there is a shared/decentralized sequencer in place.
  2. This mainly feels like a single party interested in capturing value
  3. The grant size seems too large for this
  4. Much of what is said in this proposal is already possible via the unsafe head in the gossip layer

We thank the proposer for their time putting this together and answering questions.

1 Like
19

Thank you @brockelmore for updating us on the decision and the reasoning behind it. We appreciate the time everyone took to review the Mission Request.

Two followup questions:

  1. Did the Developer Advisory Board review the original request or the updated request?
  2. How can we participate in these conversations moving forward? We are passionate about observability and ensuring equitable access to the ephemeral data layers that determine settlement – which will likely come to the forefront as the Optimism ecosystem makes progress on its decentralization roadmap. Moving forward, we would love to participate in core developer calls, office hours, etc.
20

For transparency I want to mention I spoke with the Blocknative team out of band about the rejection of this request and gave further feedback and received feedback on the process. The DAB did review the updated request, and we are thinking on ways that this process could improve as to not blindside an applicant with a rejection (in my personal view, DAB rejections should not be because of misalignment with an intent as that should be communicated before significant time is spent on an application).

1 Like