Polkassembly - Patract's Treasury proposal for Europa v0.3 (contract and runtime sandbox) #410

Patract Labs started Patract Open Platform to promote Wasm contract technology in the Polkadot community. At present, dozens of parachain and DApp projects have joined, and Europa has also been widely used in the community. 4 months ago we applied treasury proposal #27 for Europa v0.2, and 3 months ago we wrote a development report #250. In the past 3 months, we have also followed up with the updates of Pallet-Contracts and ink!.

Europa is a sandbox environment for running Pallet-Contracts. Contract developers can use the executable files compiled by Europa to debug ink! contracts. Europa is also a framework that provides a sandbox environment for Substrate Runtime. Parachain developers can implement a sandbox environment for their specific Runtime.

Europa provides minimal service components that support Runtime to package and execute extrinsics normally. Therefore, although Europa can be regarded as a node, this node does not have a consensus system, no network connection, and only retains the Runtime executor, RPC and some other necessary services. In addition, as a sandbox for Wasm contract operation, we modified Pallet-Contracts to provide more detailed debugging information, including contract execution information (information in Pallet-Contracts) and Wasm execution information. In order to facilitate debugging and add new features, Europa removed all Wasm components in Substrate, so any std code can be added to the entire system of Europa to meet experimental requirements.

Europa's future plan

v0.1(finished): Have an independent runtime environment to facilitate more subsequent expansion directions.
v0.2(finished): Modify at contract module level to provide more information。
v0.3: Improve the developement experience, strengthen collaboration with other tools, and extend the sandbox to be compatible with other runtime modules。

The original v0.3 plan about expanding Europa's compatibility with other Runtime modules has been completed in v0.1, so the focus of now work in v0.3 is to improve Europa's usablity, which is mainly divided into the following two parts:

We plan to design the UI interface for Europa and implement the following product functions:

Currently, the only UI interface that can directly operate Europa is the Polkadot apps, but apps are for chain operations rather than contract operations. We expect to show more information and elements for the wasm contracts, and improve the contract interaction experience.
1. Strengthen the functions related to the contract execution system in the contract UI section:
  1. Visualization of contract execution process information: The current important information about contract execution in Europa is directly printed to the log. Developers need to spend a lot of effort to filter out the required information in the log. Therefore, Europa will display it visually and provide some indexing functions;
  2. The relationship between the contract and related extrinsics and events: for example, when you click on the contract address, you can list the contract extrinsics related to this contract; when you click on an account address, you can list all the related extrinsics of this account, etc., this feature can assist developers in the relationship between related extrinsics, accounts and contracts;
  3. Scan all the contracts in the current Europa storage and build the relationship;
  4. Link with a Redspot project in Europa, scan the account and executed extrinsics in the Redspot configuration file, and build the relationship between contracts, accounts, and extrinsics.
2. Provide the interface and UI to operate Europa's special RPC functions and display state changes. Europa has some special functions for the sandbox environment. Currently, it can only be executed or displayed through RPC calls, such as backward of block height, forward of block height, state execution changes, etc. Operating these functions through the UI interface lowers the threshold for users.
3. Combined with Europa's workspace function, you can create and select different workspaces through the UI interface.
Similar to Ganache, Europa and UI will be made into binary packages for different platforms for distribution

Although Europa is open source, there will be a lot of trouble if you directly use the source code to compile on a different platform (such as Windows). Moreover, the compilation process is time-consuming and requires resources. Therefore, we will package Europa together with the UI into a binary executable file for distribution. Contract developers can download and execute the packaged Europa binary file directly, so that contract developers can run the node directly without worrying about how to set up the environment for running the contract. .

Detailed design of v0.3

1. The UI Interface

1.1. Visualization of contract execution process

In v0.2, the execution information of the contract in Europa will only be displayed in the log, and the structure is as follows:

1: NestedRuntime {
    ext_result: [success] ExecReturnValue { flags: 0, data:  },
    caller: d43593c715fdd31c61141abd04a99fd68...(5GrwvaEF...),
    self_account: 0144d6fc570d7bddda6f8e3614...(5C6NMXaS...),
    selector: 0x9bae9d5e,
    args: 0x40420f00000000000000000000000000,
    value: 10000000000000000,
    gas_limit: 200000000000,
    gas_left: 190018947968,
    env_trace: [
        seal_input(Some(0x9bae9d5e40420f00000000000000000000000000)),
       ...
    ],
    sandbox_result_ok: Value(I32(0,),),
    nest: [],
}

Europa UI needs to interpret and reorganize the information implemented by Europa in v0.2. Therefore, the implementation details of this development are as follows:

Europa: In this version, the information needs to be associated with the extrinsic and stored in the Europa database developed by v0.1, and provide RPC for the outside to call.
Europa UI: Return contract execution information by calling RPC with tx hash, analyze and draw it into Europa UI in a visual form (you can see all the 15 prototype diagrams)

The visualization of the contract execution process information is the most complicated part of the v0.3 product. The focus of this part is to convert the intuitionally difficult or even unreadable information originally provided by Europa into visualized information combined with the matadata information of the contract, providing developers with an intuitive display of the contract execution process.

One of the detailed prototype diagram is

Contract execution is associated with a extrinsic, which is divided into details and states. Details clearly shows the contract execution details originally in the log, and states shows the state changes involved in this extrinsic.

The top data is the overall data of the entire extrinsic, including the hash, result, block, timestamp, gas limit, gas used, extrinsic fee of the entire extrinsic, each box below represents a call, and boxes with different colors and indents represent calls For different levels, the same depth of call uses arrows of the same color. More complex details are displayed through the drop-down box of more details.

Therefore, the important meaning of this prototype diagram is to show that the "calling relationship of the contract (including the cross-contract call process)" can visually show this process, and developers can only obtain relevant information by analyzing the relationship in the log of Europa in v0.2.

The meaning of the text in the diagram is as follows:

Result: ext_result in the output of nestedRuntime. A green check is displayed for success, and a red cross is displayed for failure.
Block: the block that the extrinsic is in
TimeStamp: The timestamp of the block
From: extrinsic caller
To: extrinsic recipient
value: the token sent by the extrinsic
extrinsic fee: extrinsic fee, based on the token of the chain
Gas Limit: The maximum gas that the extrinsic can consume
Gas used: The actual gas consumed by each call, followed by a percentage that occupies the limit
more details: This part is the details of each layer of contract calls, and contains more important information.

Take the above picture as an example, the a81ac user calls the contract c650f, which is call 1. In the contract c650f, there are two more contracts called call 2 and call 3. The call depth of these two contracts is the same, so the same color arrow is used to display. Each call has a corresponding consumed gas and remaining gas. In the picture, call 2 consumes 2139256707 gas, and the remaining gas is 397860743293.

After clicking "more details":

The correct execution is as follows:

This part includes

Function: Function selector
Args: The parameters of the calling function.
trap reason: the reason for the end of this execution
env trace: host_function called during the execution of this contract

After the user uploads the ABI of the contract, he can click the "Decode parameters" button to analyze the functions and parameters called by the contract.

When the Wasm contract executes panic, a Wasm execution call stack (backtrace) message will appear:

1.2. The association between the contract, related extrinsics, events, accounts, and Redspot

The realization of this function requires Europa to be able to index extrinsics. At present, Substrate provides the function of index extrinsic by hash, which is basically about to be developed. Therefore, Europa first needs to transplant this function to the Europa framework to provide the function of index extrinsic.

Europa UI needs to write related code. After each startup, scan the block data under the current workspace from block height 0 to build the index relationship between contracts, extrinsics, and accounts.

For the association of Redspot, the corresponding interface needs to be provided on the UI, and the corresponding configuration of Redspot is parsed after the call. (Similar to the association between Ganache and Truffle)

Take the display of the contract address in Explorer as an example.

This interface not only displays the creator of the contract and the created extrinsic, but also allows the user to upload the contract ABI and directly call the related functions of the contract on this interface. At the same time, since the extrinsics sub-page lists all the related extrinsics of the contract, after the call is completed, the user can directly view the execution status of the call extrinsic initiated by the user under the sub-page, and retrieve the execution status of the historical extrinsic at any time.

1.3. Europa's special founction RPC

Europa's backward of block height, and the forward of block height. These two RPCs can currently only be called directly from the outside. So this time in Europa UI, we will combine this feature with the operability of the UI.

Take the Forks interface display in apps as an example:

When the user wants to return to the 4455515 block, he only needs to click on the block to send a request to Europa, Europa will execute the backward logic to this block, and the block information displayed on the UI will also be returned to the block.

When the user wants to forward to a higher block, there will be an input box on the UI interface. After the user enters, Europa can forward the block to the specified height.

1.4. Europa's display of state change

The current state change information of Europa only records changes in blocks (because it is directly related to StorageOverlay), and does not record Subtrie changes.

Therefore, Europa UI needs to display the status changes of the current extrinsic execution on the interface, and Europa needs to continue to develop the following 2 functions:

Europa refines the information of the state changes as a unit of extrinsic, and records it.
Europa needs to store the state changes of Subtrie.

After the development of these two functions, the UI can develop the corresponding interface for the corresponding function. In this version, only state changes similar to the hex format in Etherscan will be displayed. In future versions, we will try to display the real data after analyzing the metadata.

1.5. Europa's function of workspace

This part is similar to Ganache and provides an interface for creating and managing workspace in the UI. The main logic is to call the Europa and workspace-related command interfaces in the UI and switch the Europa workspace. Under the management of workspace, different spaces can be stored and switched under the same path.

2. Europa's binary package

Similar to Ganache, in this version Europa will be distributed together with UI into binary packages for different platforms. To make binary, we use Electron to package Europa node and UI together. Therefore, this part needs to implement the following functions:

Introduce Electron to package Europa node and UI
Solve the problem of node compilation on multiple platforms, macOS, Windows and Linux.
Produce CI's long-term support for the compilation and packaging of Electron's multi-platform nodes.

The difficulty in this part is to solve the compatibility problem between different platforms. After the completion of this part, Windows users will greatly reduce the difficulty of configuring the environment in the process of constructing the contract development environment and lower the threshold for contract developers.

Detailed timeline of v0.3

Europa is divided into the following two parts in this proposal:

Europa Node: refers to the node part of Europa, that is, the content in the original Europa repository;
Europa UI: refers to the interface part of Europa in this version. Europa-UI repository may be created or merged into the original Europa repository.

Therefore, the plan for this development is as follows:

Week1-2: Refactor part of the code in Europa Node and design the corresponding interface for Europa UI. (3 developers * 2 weeks)*

Update Europa and track to the latest Substrate version, and transplant related functions of extrisic indexing;
Persist contract execution information (Contract trace, the log information of NestedRuntime above) and provide corresponding RPC to obtain this information;
Refactor the storage module of state change information, associate the state change information with the extrisic hash, persist the data and provide the corresponding RPC;
Support storage and access to Subtrie's state change information.

Week3-5: Build Europa UI. (3 developers * 3 weeks + 1 designer * 1 week)

Europa UI framework;
Index the storage of Europa Node, and construct the relationship between accounts, extrinsics, and contracts, and make corresponding persistence;
Construct a visualization module of contract execution process information, and analyze the corresponding data obtained from Europa Node according to the contract Metadata information;
Build extrinsic, block, event, and detailed information interface of each part;
Associate each part with the RPC, storage and other parts provided by Europa Node.

Week6: Integrate and test, build the CI of Europa UI binary package, and release the binary. (3 developers * 1 week)

Cost of v0.3 (1 designer * week + 18 developers * weeks)

Operating activities: $3800 ( Rent and Devices: $200 per developer * week )
Employee payments: $35700 ($1500/$1900 per designer/developer * week)
Total Cost: $39500 and EMA30 price: $36.2 / DOT
Treasury Proposal: 1091 DOT

Verification

Directly download the binary packages of Windows, macOS and Ubuntu from Europa Release, which can be run directly on the corresponding platform.
After deploying any contract through Europa and calling it through a extrinsic, the functions planned in the prototype diagram are presented:
1. In the associated account, you can click to jump between extrinsics and contracts. For example, click on an account to list the contracts deployed under this account and initiate invocation extrinsics, and click on the corresponding extrinsic or contract to jump to the corresponding detailed interface;
2. Click on a extrinsic that deploys or invokes a contract, and the detailed information of the contract execution will be listed in a visual form; click on the state interface of the extrinsic, and the state information changed during the execution of the extrinsic will be listed;
3. On the main page displaying block information, you can click on a block and roll back to that block, or enter a block height to forward the height of the Europa block to that block;
You can create and switch between different workspaces through the workspace interface;
After Europa is closed and restarted, the relationship between accounts, extrinsics and contracts before closing will be reconstructed based on the information in Europa's workspace.