Introducing Arweave’s Tech Ecosystem (Part 2)

This post is based on the speech that outprog, the everFinance’s founder who proposes the SCP (Storage-based Consensus Paradigm) theory, gave on BeWater Live. It is intended to give you the whole picture of Arweave’s tech ecosystem by delving into Arweave, SmartWeave and SCP, as well as the two-layer model that integrates PoW and PoS.

This post is divided into three parts, each of which centers on the following content:

1. How Arweave Works

- Why is perpetual storage economically viable

- How does Arweave ensure miners store all the data

- Blockweave technology

- Implicit incentive mechanism

2. SmartWeave and SCP

- SmartWeave

- Storage-based Consensus Paradigm

- The Web3 Era of Infinite Scaling

- everPay Architecture

3. The Two-layer model that integrates Permanent Storage and PoS

- The Design of KYVE

- The Design of Koii that Integrates Permanent Storage and Pos

- Conclusion

SmartWeave and SCP

This is the second part of “Introducing Arweave’s Tech Ecosystem”. This post will discuss perpetual storage and consensus computation.

1. SmartWeave

When I first learned about Arweave last July (2020), I was aware that it would be a game-changing underlying blockchain. It occurred to me that a blockchain or cloud storage designed for perpetual storage could be considered as a tape of a Turing machine.

As we all know, a Turing machine is composed of a state machine and an infinitely long tape. Arweave is ideal for serving as a tape. We can write a program that is able to perform any computation and store it in Arweave with all the data.

For example, when you create a financial statement using Excel on your personal computer and send it to others, they cannot confirm whether your financial statement is authentic.

However, if the hard drives of the computer are replaced with a storage blockchain, the financial statement can be trusted with all the data stored on-chain and computations done by independent clients in a distributed way.

Hence, we believe that as long as the data stored on-chain is trustable, computations can be performed in any compute unit and the results computed on immutable data via a deterministic program are unique and trustable.

In Bitcoin, the ledger is created using the PoW mechanism, which helps build a consensus. However, with Arweave using hard drives instead, each operation and its steps performed on your personal computer will be recorded on-chain.

Hence, the current state is always traceable and verifiable. Although it seems to be a radically new model, it is actually a classical model that returns to the essence of computers. It ensures that every file is traceable and immutable with no need to perform computations on-chain like Ethereum.

As long as a deterministic program is uploaded to Arweave, it can be downloaded by anyone and can automatically load the data on-chain to perform computations. In this case, any computation result can be considered as an objective fact or a consensus. So how is SmartWeave able to make all this possible?

SmartWeave is a virtual machine that runs off-chain. It can read the running code of applications, load all the input parameters from Arweave, and then locally execute each of the transactions required. After that, Smartweave syncs up with the latest Arweave block so that it can generate the latest finalized state.

The code of smart contracts is not executed on-chain at all.

Arweave can guarantee the authenticity of the code stored in an Arweave transaction, which can be a token contract, for example. The state of the contract and all the relevant operations are the outcome of users sending Arweave transactions.

Although it sounds perfect, SmartWeave has a weak spot for its inability to avoid DDoS attacks. A DDoS attack means that a hacker sends a massive amount of invalid transaction data that will not be executed successfully.

Since SmartWeave cannot tell which data is valid before executing transactions, the SmartWeave clients have to keep loading the data to compute the results. If there are multiple gigabytes of invalid transactions, SmartWeave will suffer from low availability, but its security and consensus remain unaffected.

2. Storage-based Consensus Paradigm

Figure 1 shows the Bitcoin model. Before you make a transfer using Bitcoin, you need to run a full node or SPV node on your personal computer to recompute Bitcoin’s ledger.

However, since 2017, the proxy model adopted by MetaMask and imtoken has gained popularity. In this model, application operators deploy full nodes on Ethereum and Bitcoin for users to request data and send transactions.

Similar to the Figure 1 Bitcoin model, the SmartWeave model requires users to perform computations and send transactions on their own. However, in the proxy model shown in Figure 2, proxies like MetaMask can display a false balance for users, but they cannot change the consensus.

Whoever performs a computation on the consensus data through a specific program must obtain an identical and unique result. Anyone who runs a node is unlikely to obtain a false state by computation. The same goes for SmartWeave — all the calculated results are finalized.

Taking one step further, we can discover that users no longer need to interact with blockchains on their own. Instead, they only need to access the application clients that interact with blockchains.

Take SmartWeave for example, it depends on an application client, which can be run on the provider’s server, to perform off-chain computations. A perpetual storage blockchain allows the server to be a deterministic and traceable program with the same characteristics as a blockchain.

1) Relative Consensus and Absolute Consensus

Why do users no longer need to access blockchains directly? To find out the answer, you need to understand the concepts of relative consensus and absolute consensus.

For example, an eth node shown on the left is deployed by imtoken. When you send a transaction using imToken, the transaction is relayed to the eth node via the imToken gateway.

In this case, you completely trust a third party that can forge data, which is called relative consensus. On the contrary, if you deploy a node on your own to execute all the transactions without relying on any third party, this is called absolute consensus.

The proxy model allows users to use many DeFi products, which usually provide an APR for yield mining. To obtain an actual and reliable APR, users must perform computations on their own.

When users rely on centralized nodes to provide slightly abstract data like APR, the third-party servers can offer a false APR to them, but they cannot change the consensus on transfers. To go one step further, third-party nodes can be trusted to compute the basic information of transfers like balance. They simply need to provide a user-friendly API.

So we’ve built the Trusted APP model, in which a Trusted App takes the place of blockchains to interact with users. Decoupling users from blockchains is the key to lowering the user threshold and increasing TPS.

2) Trusted APP

This model has the following advantages:

• Composability

The Trusted APP offers composability on the basis of data partitioning. When composing with each other, Application A can download the data of Application B from the storage blockchain and verify it according to the business rules of Application B, thus achieving multi-ledger verification.

It allows us to only integrate the data of applications that we care about for verification. This model greatly improves the composability of traditional businesses using a blockchain. It reminds me of the days when I worked at a bank.

During the period, I was responsible for multi-party account checking and found it painful to do it in a traditional way. Everyday I was sent a text file to check my ledger with the third-party ledger. Such a semi-manual method is error-prone. If any error occurs, reversal is required.

However, with Arweave storing all the data, the multi-party account checking can be done via an automated program. This can help create a ledger that is very close to Bitoin’s. All the DeFi and traditional applications can be composed in this way so that account checking can be done per block instead of per day.

• Low development barriers

The storage-based consensus paradigm greatly reduces barriers to development. As I once demonstrated, it is quite simple to write a SmartWeave contract with custom states in JavaScript.

Imagine that the balance state of an application is state.balance, the registration state of an email address is state.registered, and the state of a perpetually stored file is state.file_id. The paradigm makes development quite simple and flexible without any limitation to programming languages.

The PST tokens on Arweave conform to a specific standard like ERC20 tokens and have some general APIs for states. However, there is no restriction on the development of Web2 and Web3 applications so that developers can flexibly utilize the advantages of SmartWeave.

More importantly, you do not have to write smart contracts in JavaScript or use SmartWeave. As far as I am concerned, SmartWeave is just one of the examples of the storage-based consensus paradigm.

With its perpetually trustable storage supporting consensus computation, Arweave becomes a tape of a Turing machine so that you can build any Turing-complete application. And you can go even further by replacing SmartWeave with your own back-end program, which will have the outstanding properties of a blockchain.

Given that SmartWeave can’t prevent DDoS attacks, we have transformed its model by using a high-performance back-end server instead to receive users’ transactions. Despite its resistance to DDoS attacks, the improved model has a disadvantage because the independent back-end server can filter and censor users’ transactions.

However, the back-end server is unable to change the objective states because all the transactions included into the chain are signed by users themselves. The improved model can resist DDoS attacks that SmartWeave is vulnerable to, while offering the same development mode as Web2.

• Infinite TPS

Besides, the TPS of applications is infinite because the server can roll up a batch of 10,000 or 100,000 transactions in a single block and then include it into Arweave. In this process, a temporary state will be waiting in line.

A batch of transaction data is rolled up into Arweave every 5 to 10 minutes so that the corresponding state will be finalized. No matter what server your program runs on, you can easily build a trustful Web3 application by using Arweave as perpetual storage.

To make this model more abstract, you can build a Web3 application by combining a storage blockchain with AWS or Google Cloud. With Arweave guaranteeing the immutability and traceability of input data, a deterministic business program can ensure the uniqueness and finality of output data.

How can we apply this model to DeFi? The API of a Trusted APP offers real-time data verification and inclusion into the chain. Multi-party ledger verification is required in the DeFi field. When integrating a token or a lending protocol like Compound, a DEX needs to download their business programs and historical data from Arweave.

For example, it can integrate the JavaScript code of SmartWeave for calculation and data verification. You can write your own program in any programming language like Golang or Rust to convert the deterministic rules of other applications into the language that you use. And you can integrate these applications into your own one simply by using their SDKs.

3) The Web3 Era of Infinite Scaling

The storage-based consensus paradigm is expected to create a Web3 era of infinite scaling. As shown in the figure above, on the left is the token payment and settlement protocol everPay, on the right is an AMM program, and at the bottom is SNX.

All these applications store their business data on a storage blockchain with their consensus data computed by detectors. They communicate with each other via HTTP in real time, and the communication results can be verified and reversed afterwards. Since everPay can obtain all the data of the AMM and check the AMM’s consensus with a specialized program, it can compute the final state of the AMM. As long as the latest data of the AMM is rolled up into the storage blockchain, the three applications will obtain the same result when computing the AMM’s state.

everPay can tell if other applications are deterministic and fraudulent. The applications interact with each other via traditional Web2 APIs in real time. Once the data is included into the chain, the entire state will be finalized.

Hence, the temporary state generated during interaction will also be finalized. Besides, the storage-based consensus paradigm does not require each of the applications to use the same storage blockchain.

To be specific, an application that uses Arweave can still compose with another application that uses a different storage mode. For example, an AMM uses a storage blockchain different from everPay. If everPay wants to verify the AMM, it can read its data directly from the storage blockchain used by the AMM for computation and verification. That is how composability is realized.

3. everpay

everPay is a dApp of both Ethereum and Arweave. When using everPay, we can directly connect to it via the Ethereum wallet MetaMask, or the Arweave wallet ArConnect. In the future, everPay will add support for more blockchains such as Solana and Near. The flexibility of off-chain computation allows us to build various cryptographic implementations with Golang. For now, everPay is more like a Layer 2 application that offers real-time token transfer and a bridge between Arweave and Ethereum.

Before using everPay for the first time, you need to make a deposit. In everPay, all the assets are managed by a DAO, whose members have a multisig contract on Ethereum and Arweave respectively.

When you make a deposit to an address managed by the DAO, the transaction will be sent to everPay, which will check if the transaction is confirmed by 15 blocks on Arweave. If the transaction is validated, the deposit will be mapped 1:1 to everPay. It means that the deposit address will hold the same amount of newly-created tokens, which can be controlled by the original wallet.

When you transfer, for example, 1 USDC token, the receiver address can be an ETH or AR address. Any ETH or AR address is considered a valid everPay account. When making a transfer, you need to sign the transaction via MetaMask.

In the figure above, you can see the transfer amount and the fee of the transaction, which will be directly sent to the server. If you switch to the receiver wallet on Arweave and connect it to everPay, you will find its address is totally different from the ETH address. And you will see the transfer transaction in the transaction history of your everPay account.

By performing computations off-chain, everPay offers extremely high performance. Similar to PayPal and Stripe, everPay can utilize various back-end programs to build high-performance applications. However, all the everPay transaction data will be rolled up into Arweave so that the everPay ledger has the same characteristics as a blockchain.

everPay Architecture

The entire architecture of everPay is shown in the following figure. It contains three roles:

Coordinator:

The coordinator is analogous to the sequencer in a zkRollup. All the users’ transactions are sent to the coordinator for real-time computation on a first-come, first executed basis, so as to get finalized.

Since users’ transactions are confirmed as soon as they are received by the coordinator, there is no such thing as a pending transaction in everPay. If a transaction is sent successfully, it will be added to a transaction pool and wait for being included into the chain; if not, it will return an error to the front end and will not be added into the pool. The transactions in the pool are included into Arweave by batches every 5 to 10 minutes.

The business program run by the coordinator is consistent with those run by watchmen and detectors. Each role loads data alone from Arweave for independent computation and verification.

Detector:

Similar to Infra, a detector offers all the transfer and balance information after it is deployed. Users can obtain the ultimate state of the ledger from detectors.

Watchman:

A watchman takes charge of multisig governance in cross-chain interactions. It can be considered as a detector in control of a private key. When a watchman detects any withdrawal data on Arweave, it will create a withdrawal proposal. If you want to withdraw tokens native to Ethereum, watchmen will verify your transaction and sign a withdrawal proposal. Your transaction will be executed on Ethereum or Arweave only when it is signed by over 51% watchmen.

The ledger synchronization among watchmen, detectors and the coordinator completely depends on Arweave instead of P2P communication. All the roles will verify the entire ledger on their own. After verification, watchmen need to sign a proposal.

Building applications on top of the SCP with Arweave as storage infrastructure will become a real trend in the Web3 era. everPay is a typical Web3 application that offers nearly the same TPS as a traditional Web2 application and operates as a DAO.

Take mirror.xyz as an example. We published the everPay lightpaper on Mirror. Although Mirror runs on a centralized server, all the file data is stored in Arweave, while the corresponding NFTs are minted on Ethereum, which is completely in conformity with the SCP model.

Perpetual storage is the key to the SCP. By using a blockchain like Ethereum as the back end, your applications do not need to run completely on-chain or in a fully decentralized way. In fact, decentralized storage allows users to store their files permanently via a centralized service instead of sending a transaction on their own.

Furthermore, asset transfer might also benefit from the SCP model. By expanding the value flow, everPay does not require users to send transactions on their own. It allows asset mapping cross-chain, making blockchains on longer difficult to use.

everPay currently supports Arweave and Ethereum, and will be able to integrate any blockchain in the future. everPay offers real-time transfers and almost the same user experience as traditional transfers.

It will add support for smart contract accounts, which are costly on-chain, but are much cheaper off-chain. everPay’s smart contract accounts will provide decentralized security and an optimal user experience at a relatively low cost in consensus building.