Authors: Pranav Garimidi, Joachim Neu, Max Resnick, a16z crypto researchers; Source: a16z crypto; Compiler: Shaw Golden Finance

Today’s blockchain can confidently claim that it has the processing power needed to compete with existing financial infrastructure. The current production environment system can handle tens of thousands of transactions per second, and performance will be improved by orders of magnitude in the future.

However, financial applications require predictability beyond pure throughput. When a transaction is initiated - whether it is a trade, an auction, or an option exercise - being able to reliably ensure the time when the transaction is confirmed on-chain is critical to the normal functioning of the financial system. If transactions face unpredictable delays (whether due to malicious interference or accidental circumstances), a large number of applications will not function properly. For on-chain financial applications to be competitive, the blockchain must provide short-term transaction on-chain guarantees: as long as a valid transaction is submitted to the network, it is guaranteed to be packaged and on-chain as soon as possible.

Take the on-chain order book as an example. An efficient order book requires market makers to provide liquidity by placing continuous orders to buy and sell assets. The core problem faced by market makers is to minimize the bid-ask spread (the difference between the buying price and the selling price) while avoiding the risk of adverse selection caused by the quote being disconnected from the market. To do this, market makers must constantly update orders to reflect the latest market conditions. For example, when an announcement by the Federal Reserve causes a large fluctuation in asset prices, market makers need to immediately update orders to the new price. In this case, if the transaction used by the market maker to update the order cannot be uploaded to the chain immediately, the arbitrageur will be executed at the outdated price, causing it to suffer losses. Market makers can only be forced to widen price spreads to reduce such risks, which in turn causes on-chain trading platforms to become less competitive.

The predictable transaction on-chain mechanism can provide reliable guarantees for market makers, allowing them to quickly respond to off-chain events and maintain the efficient operation of the on-chain market.

The gap between current situation and needs

Currently, existing public chains can only provide relatively reliable final chain guarantee on the scale of a few seconds. This kind of protection is sufficient for payment applications, but it is far from sufficient for a large number of financial applications that require market participants to respond to information in real time. Take the above order book scenario as an example: For market makers, the guarantee of “on-chain within seconds” is meaningless if the arbitrageur’s transactions can be prioritized and packaged into earlier blocks. Without strong on-chain guarantees, market makers can only hedge against higher adverse selection risks by widening price differences and providing users with worse quotes. This would make on-chain transactions less attractive than other trading venues with stronger guarantees.

For blockchain to truly realize its vision of becoming a modern infrastructure for capital markets, developers must solve these problems to allow high-value applications such as order books to flourish on the chain.

What is so difficult about predictability?

It is extremely difficult to strengthen transaction on-chain guarantees on existing public chains to support such financial scenarios. Some current protocols rely on a single node (the "block producer") to decide which transactions can be packaged within a specified time. Although this design reduces the engineering difficulty of building a high-performance public chain, it also creates a potential economic monopoly point - block-producing nodes can use this to capture value. Generally speaking, during the window period during which a node is selected as a block producer, it has absolute control over which transactions are included in the block.

For public chains that carry various financial activities, block-producing nodes are in a privileged position. If the node refuses to package a transaction, the user's only option is to wait for the next block-producing node that is willing to package the transaction. In a permissionless network, block-producing nodes naturally have an incentive to capture value, which is commonly known as miner extractable value (MEV). MEV is much more than a sandwich attack on AMM transactions. Even if the block-producing node only delays transaction packaging for tens of milliseconds, it can still make huge profits and significantly reduce the operating efficiency of the underlying application. If the order book only prioritizes the transactions of some traders, it is unfair to all other participants. In extreme cases, malicious behavior by block-producing nodes may even cause traders to abandon the platform completely.

For example: The Federal Reserve announced a rate hike, and the price of ETH instantly fell by 5%. All market makers on the order book hurriedly canceled the original pending orders and placed new orders at the new price. At the same time, all arbitrageurs also submitted trades trying to sell ETH on expired orders. If the order book is running on a protocol designed with a single block producer, this node will have a lot of power. It can directly choose to review the cancellation transactions of all market makers, allowing arbitrageurs to earn huge profits; or it can not completely review the cancellation transactions, but delay the processing until the arbitrageur transactions are uploaded to the chain before releasing them; the block-producing node can even directly insert its own arbitrage transactions to fully profit from the price deviation.

Two core demands: censorship resistance and transaction concealment

Faced with this advantageous position, the active participation of market makers becomes unprofitable - they can be exploited whenever prices fluctuate. The problem essentially stems from the excessive privileges of block-producing nodes in two aspects: block-producing nodes can review the transactions of any trader; block-producing nodes can view other people's transactions and submit their own response transactions accordingly. Any of these problems can have catastrophic consequences.

Examples

We can clearly illustrate the problem through a case. Suppose there are two bidders in an auction: Alice and Bob, and Bob happens to be the block producer of the block where the auction is located. (Only two bidders are set for ease of explanation. This logic holds true for any number of bidders.)

The auction accepts bids during block generation, with the time interval set from t=0 to t=1. Alice submits a bid bA at time tA, and Bob submits a bid bB at time tB (tB > tA). Since Bob is the block producer of the block, he can always ensure that he bids last. Both Alice and Bob can read the continuously updated real price of an asset (such as the middle price of a centralized exchange). Suppose the price at time t is pt. We assume that at any time t, the asset price expected by both parties at the end of the auction (t=1) is equal to the current pt. The rules of the auction are simple: the one with the higher bid wins and the transaction is completed based on its own bid.

The need for censorship resistance

Now let’s look at what will happen if Bob takes advantage of the block-producing node. If Bob can review Alice's bid, the auction mechanism will immediately fail. He only needs to quote a very low price to be sure of winning, because other quotes are blocked. This will result in the final revenue from the auction being close to zero.

The necessity of transaction concealment

A more complicated situation is: Bob cannot directly review Alice's bid, but he can see her bid before he bids. At this time, Bob will adopt a simple strategy: when bidding, determine whether the current price ptB is higher than bA. If so, he bids a price slightly higher than bA; if not, he gives up the bid.

Through this strategy, Bob exposes Alice to continuous adverse selection. Alice can only win if her quote is higher than the expected value of the asset after the price is updated. Whenever she won an auction, she expected to lose money and eventually stopped bidding at all. When all competitors leave, Bob can bid a very low price to win the auction, causing the auction proceeds to return to zero.

The core conclusion of this case is:The length of the auction does not matter. As long as Bob can review Alice's bid, or see Alice's bid before he bids, the auction is doomed to fail.

This logic also applies to all scenarios of high-frequency asset trading, including spot trading, perpetual contracts, derivatives exchanges, etc.: If there is a block-producing node with the power of Bob in the case, the entire market mechanism will completely collapse. For on-chain products that serve such scenarios to be feasible, block-producing nodes must not be given such privileges.

How do these problems arise in reality today?

The above description paints a bleak picture for any on-chain transaction using a permissionless single block producer protocol. However, trading volumes on decentralized exchanges (DEX) are still performing well on many of these protocols. What’s the reason behind this?

In practice, there are two forces that jointly offset the above problems:

• Block-producing nodes have not completely abused their economic privileges, because they usually have huge interests tied to the success of the underlying public chain;

• Apps have built workarounds to reduce their vulnerability to this type of issue.

Although these two factors have allowed decentralized finance (DeFi) to operate so far, in the long term, they are not enough to allow the on-chain market to truly compete with the off-chain traditional market.

On a public chain with considerable economic activity, a large number of tokens need to be pledged to qualify for block production. Therefore, block producers either hold a large amount of collateral themselves, or have enough reputation to allow other token holders to delegate their pledge rights to them. In either case, large node operators are typically public entities with reputations at risk. In addition to reputation, the collateral they hold also means that they are financially motivated to promote the development of the public chain. Because of this, we have rarely seen block producers completely abusing market power as described above - but this does not mean that these problems do not exist.

On the one hand, relying on social pressure and long-term interest demands to constrain the goodwill of node operators is not a reliable cornerstone of the future financial system. As the scale of financial activities on the chain expands, the potential profit margins of block-producing nodes will also increase simultaneously. The greater this potential, the more difficult it will be to constrain block-producing nodes at the social level and make them go against short-term interests.

On the other hand, the extent to which block producers abuse market power is on a continuum, ranging from mild behavior to complete market destruction. Node operators can unilaterally and gradually experiment and use their power to obtain higher profits. When some operators push the boundaries of acceptable behavior, other nodes quickly follow suit. The behavior of a single node may seem to have limited impact, but when everyone begins to follow suit, the impact will be very obvious.

Perhaps the best example of this phenomenon is the timing game: the block-producing node will delay publishing the block as much as possible while ensuring that the block is still valid in the protocol, so as to earn higher rewards. This will lead to a longer block generation time, and if the node is too aggressive, blocks may even be skipped. Although the profitability of such strategies is well known, block producers initially chose not to participate in such games, mainly to maintain the image of good managers of the public chain. However, this social balance is fragile. Once a node operator starts using such strategies to achieve higher profits with impunity, other nodes will quickly follow suit.

The timing game is just one example of how block producers can increase their profits without completely abusing their power. There are many other ways they can improve their own returns at the expense of their applications. Taken individually, these methods may still be within the affordable range of the application, but they will eventually reach a critical point where the on-chain operating costs exceed the benefits.

Another factor that keeps DeFi functioning properly is that applications move their core logic off-chain and only put the results on-chain. For example, all protocols that need to execute auctions quickly will choose to do so off-chain. Such applications usually run the required mechanisms on a set of permissioned nodes to avoid problems caused by malicious block-producing nodes. For example, UniswapX runs the Dutch auction used to match transactions off-chain on the Ethereum mainnet, and Cowswap similarly runs batch auctions off-chain.

Although this method can ensure the operation of applications, it puts the underlying public chain and the core value proposition built on it into a precarious situation. If the execution logic of the application is all off-chain, the underlying public chain will be reduced to a simple settlement layer. One of the core advantages of DeFi is composability, and in a world where all execution occurs off-chain, these applications are naturally in a disconnected environment. Relying on off-chain execution will also add new prerequisites to the trust model of these applications: applications not only need the underlying public chain to operate normally, but also must rely on the availability of off-chain infrastructure to work properly.

How to achieve predictability

To solve these problems, the protocol needs to meet two major characteristics:Stable transaction on-chain and ordering rules, and privacy protection before transaction confirmation.

Core requirement one: censorship resistance

We summarize the first feature as short-term censorship resistance. If a protocol is short-term censorship-resistant, then any transaction that reaches an honest node is guaranteed to be included in the next available block.

Short-term censorship resistance: Any valid transaction that is delivered to any honest node on time will definitely be included in the next block.

More precisely, we assume that the protocol runs on a fixed clock, with blocks generated at a fixed time, e.g. one every 100 milliseconds. Then we need this guarantee: if a transaction reaches an honest node at 250 milliseconds, it will definitely be included in a block generated at 300 milliseconds. An attacker should not have the power to filter transactions, selectively package or eliminate certain transactions at will. The core idea of ​​this definition is that users and applications should have a highly reliable way for transactions to be successfully uploaded to the chain at any time. A transaction should not be unable to be uploaded to the chain just because a single node loses packets - whether due to malicious intent or malfunction.

Although this definition requires that transactions delivered to any honest node be guaranteed to be on-chain, in practice, the overhead of achieving this may be too high. The core significance is that the protocol should be stable enough so that the entrance to the transaction chain is highly predictable and the logic is simple and easy to understand. The permissionless single-block-producing node protocol obviously does not meet this characteristic, because if the only block-producing node at a certain moment acts evil, there will be no other way for the transaction to be uploaded to the chain. However, even if only a group of four nodes in total can ensure that transactions are packaged in each time period, it will greatly increase the transaction on-chain options for users and applications. In order for your application to thrive stably, it's worth sacrificing some performance for reliability. Finding the right balance between robustness and performance still requires further research, but the guarantees provided by existing protocols are far from sufficient.

Once the protocol can ensure that transactions are on-chain, the ordering problem will be solved. The protocol can adopt any deterministic sorting rule to ensure consistent sorting results. The simplest solution is to sort by priority fee, or allow the application to flexibly order transactions that interact with its state. The optimal way to order transactions is still an active research direction, but in any case, ordering rules only make sense if transactions can be successfully uploaded to the chain.

Core requirement two: information concealment

After achieving short-term censorship resistance,another key feature that the protocol needs to meet is a privacy protection capability we call concealment.

Concealment: Before the transaction is finally confirmed on the chain by the protocol, no other party can obtain any information about the transaction except the node receiving the transaction.

Protocols that satisfy concealment can allow receiving nodes to view all transactions submitted to them in clear text, but require the rest of the protocol to know nothing about the contents of the transactions until consensus is completed and the order of the transactions in the final log is determined. For example, the protocol can use time-lock encryption to make the entire contents of the block invisible before a certain deadline; or use threshold encryption to decrypt the block only after the committee confirms that the block is irreversible.

This means that a node may abuse the transaction information submitted to it, but other nodes in the network will not know the content to be confirmed before the consensus ends. When the transaction information is disclosed to the rest of the network nodes, the transaction has been sorted and confirmed, so other participants cannot conduct preemptive transactions. In order for this definition to be practical, transactions must be completed by multiple nodes at any time.

The reason why we did not adopt a stronger definition of privacy - that is, only the user knows the transaction content before the transaction is confirmed (such as the encrypted transaction pool solution) - is because the protocol requires a certain link to serve as a spam transaction filter. If the transaction content is completely hidden from the entire network, the network cannot distinguish spam transactions from valid transactions. The only solution is to keep some non-hidden metadata in the transaction, such as the payment address, and the fee will be deducted regardless of whether the transaction is valid or not. However, this type of metadata may still reveal enough information to allow an attacker to take advantage of it. Therefore, we prefer to design such that the complete transaction is visible to a single node and invisible to other nodes in the network. However, this also means that for this feature to be effective, users must have at least one honest node as the entry point for transactions to be uploaded to the chain in each period.

Conclusion

A protocol with both short-term censorship resistance and information concealment is an ideal underlying foundation for building financial applications. Returning to our example of running an auction on-chain, these two features directly address the issue of Bob potentially causing a market crash. Bob can neither review Alice's bid nor use Alice's bid to guide his own bid, precisely solving problems 1) and 2) in the previous example.

With the guarantee of short-term censorship resistance, any user who submits a transaction - whether it is a transaction or an auction bid - can ensure that the transaction is immediately packaged and uploaded to the chain. Market makers can update orders, bidders can place bids quickly, and liquidation operations can be executed efficiently. Users can be confident that any action they initiate will be performed immediately. This will enable a new generation of low-latency real-world financial applications to be built entirely on-chain. For blockchain to truly compete with, or even surpass, the performance of existing financial infrastructure, we need to solve much more than just throughput issues.