History and notifications in EOSIO blockchain

These are quite common questions in EOSIO developers chat and in other places:

• How do I find all recent transactions relevant to my account?
• How do I get notified of new transactions?

Back in early 2018 when EOSIO sofware was released, it included history_plugin and history_api_plugin which allowed storing and searching the transaction history.

A problem that was difficult to foresee at that time, is that the history plugin stores all transaction traces in the same shared-memory database (chainbase) where nodeos keeps the blockchain state. This is a large file-mapped shared memory segment where data is written and read randomly. Even without the history plugin, running an EOS mainnet node requires that this shared memory file resides on directly-attached SSD (another option would be ramdisk, but that requires lots of RAM, plus a backup automation). The full history of EOS comprises of few billion transactions, so it becomes a big and expensive challenge to maintain this data in a shared-memory file. It requires big and expensive server hardware, and in case of a crash, replay can take significant time (longer than a month for a full EOS replay).

In addition, the history plugin is recording public keys and their association with accounts. So, software like Scatter needs that information in order to find accounts managed by a private key in the wallet.

As of today, several options are available for serving the history requests.

Legacy history_plugin

Although officially deprecated by Block.one, the original history plugin is still widely in use. For a small test network, it’s the easiest and fastest way to get the list of transactions for an account, and services like local.bloks.io work with it out of the box.

Only a few EOS block producers are maintaining the full EOS history with history plugin. Many public service providers decided to wrap it up because hardware costs are becoming unaffordable.

Greymass team has come up with an interim solution, the Light History plugin. It’s a drop-in replacement for history_plugin, but it keeps the history data in a separate shared memory file, different from node state, and also it only stores a given number of latest transactions for each account. Older transaction traces are removed from the database, and the public service hands requests off to a full history node. This saves a great deal of CPU resource on the full history node, and allows reducing the hardware and operational costs.

dfuse

dfuse is a service provided by EOS Canada. The software is not open-source, but the service is used by many dApps and is proven to be reliable and flexible. It offers search queries on blockchain history, and subscription service in order to listen for specific events in real time. The free plan allows 250K documents per month, and there are paid plans for higher demands. EDIT: dfuse is now available as open source.

Hyperion

Hyperion is an open-source software solution offered by EOS Rio. It takes state_history_plugin (explained below) input and stores the blockchain history in Elastic Search database. The HTTP API of Hyperion is resembling that of the legacy history API, so it’s easy to adapt applications to use it as a replacement. Several BP provide public Hyperion endpoints for EOS and other blockchains.

Hardware requirements are much more affordable than those for a full history plugin node. Also the job can be split between several physical machines. A server with 64GB RAM and 4TB disk storage would suit to hold the whole EOS history as of August 2019. EDIT: As of September 2020, 128GB RAM and 20TB or more storage is recommended.

state_history_plugin

State history plugin for nodeos is part of EOSIO distribution. It stores all transaction traces and table deltas in compressed archive files. Also it provides a Websocket interface for querying the traces and deltas for a specific block interval. If the end block number is in the future, the plugin delivers new traces as they become available in real time. It has also an option to deliver only traces from irreversible blocks.

The data that is exported via this Websocket interface is in binary form. The plugin is designed to spend as little as possible CPU time on interpreting the data, so it’s the job of the client to decode the output and process into a usable form. Action arguments are delivered in serialized form, in the same binary form as they are submitted to the smart contract. The same applies to table deltas: the plugin does not try to interpret the contract table contents, it just sends you modified rows as they are stored in server memory.

So, in order to be able to interpret its output, the client software has to have copies of ABI that the smart contracts are publishing. The ABI is delivered in the same binary form as part of contract data changes when setabi action is executed. As smart contracts and ABI get updated, the client software needs to maintain relevant revisions of ABI in order to be able to interpret the history data.

As of today, several open-source software projects are available for reading the plugin’s data:

• history-tools by Block.one is a set of tools that store the state history data in a database and provide API for retrieving it. As of today, the software is in development and not ready for production.
• Hyperion by EOS Rio implements the whole interpreter of state history in JavaScript. It stores ABI revisions in its database and decodes the traces into JSON documents which are sent to ElasticSearch for storing and indexing.
• Chronicle by cc32d9 is a C++ program that serves as a bridge between the state history plugin and consumer. It stores ABI revisions in its own database (using the same chainbase software as used in nodeos), and it expects the consumer to open a Websocket server where it connects and pushes a stream of JSON objects. The development was sponsored by multiple BP and wallet teams, and also by Telos WPS.
• eosio-statereceiver by EOSDAC is yet another interpreter of state history data written in JavaScript.

One of recent challenges is that nodeos version 1.8 changed the format of state history, so the client software needs to be adapted accordingly. Also the transaction traces are in a different format: 1.7 and earlier delivered inline traces as hierarchical objects, whereas 1.8 presents them as a linear array.

My scripts for Chronicle

I wrote a number of Perl scripts that take Chrionicle output for further processing. They can be used as a reference if you need to develop your own processor:

• Light API is a production-ready piece of software. It stores current token balances and account security information in MySQL database, and provides an HTTP API for retrieving the data. It also provides a way to find all accounts controlled by a public key. The API is available for public, and is in use by Scatter, Bloks.io, and few others.
• eosio-analytics is storing all token transfers and voting events in MySQL database.
• eosio_chaindb is similar to analytics script, with a slightly different purpose. It stores all token transfers and history of all token balances (including REX and staking) in MySQL database for multiple blockchains.
• dumper scripts are used mainly for testing the Chronicle itself. They can also be used to produce a sample output, so that it’s easier to design the consumer software.
• history indexer POC is work in progress. Details and purpose are described in my fundraising article. The goal is to make a replacement for the history plugin, and the difference from Hyperion is that it doesn’t store transaction traces in the database, but keeps only an index of traces.

Iris client by CALEOS

Iris is an open-source project by CALEOS, with the goal to build an alternative to dfuse.

Users can subscribe to real-time events for their accounts of interest and receive events with transaction traces.

EOS USA is preparing their endpoints to serve Iris for several networks, such as Telos, EOS, WAX, and others.

Other nodeos plugins

There’s a number of plugins for nodeos that are exporting transaction traces in JSON directly to some external receiver. There are two fundamental problems with them:

1. nodeos is already quite busy with processing the network (for example, a full replay of EOS Mainnet would take several weeks), and these plugins add more work by producing JSON data within the same main thread, taking the CPU time from the actual blockchain processing. This slows down the whole process significantly.
2. Microforks are typical and quite frequent in EOSIO blockchain. Mostly they are caused by time zone difference between block producers when they hand over the schedule. Also a BP may leave two producer nodes signing blocks concurrently, and that leads to microforks as well. Most of known plugins for nodeos are either handling the forks incorrectly or not handling them at all. As far as I know, only state_history_plugin and my own zmq_plugin are reacting to forks in a correct way: they inform the receiver about the fork, and roll back their internal state to the point of forking.

mongo_db_plugin is part of EOSIO distribution, but deprecated from using. It’s recommended to run in read-mode = read-only because it’s not handling speculative transactions well. It is also known for crashes and data corruption.

My own zmq_plugin is what I was using in production before state history and Chronicle became available. It’s stable, and it handles forks correctly by notifying the receiver about them. But the nature of ZMQ socket does not guarantee 100% delivery, and messages may get dropped in the beginning and at the end of the ZMQ socket session. It’s still usable as a lightweight notification mechanism, especially with its recent whitelist function. Not recommended as a history solution because of high CPU overhead and probability of data loss.

There’s also a number of third-party plugins, such as one by TokenPocket exporting into Apache Kafka queue, or the one by Attic Labs exporting into Apache Cassandra database. But they are subject to the issues outlined above: performance and lack of fork handling.

My recommendation would be to stick to the state history plugin and build the applications around it.

Conclusion

The original history plugin is phasing out, and there are several alternatives, based on top of state history plugin. Many dApp developers are still looking for possibilities to utilize the old history API, although new alternatives are offering much more possibilities and are future-proof.

In many cases a dApp does not really need the history, but it needs notifications about new transactions. The new tools are offering that already.

As of today, most of these new tools require building a dedicated infrastructure. But there are solutions and offerings in development that will allow sharing and re-using the infrastructure for multiple applications.

If you want to add a new project to this article, do not hesitate to contact me at @cc32d9 on Telegram or via email at cc32d9@gmail.com