WAX blockchain node performance challenges

The WAX blockchain has grown in usage and size pretty dramatically. On May 8th 2021, 307k unique accounts have executed 51 million actions. On February 8th, it was 18k and 6 million, correspondingly.

Back in February, a typical nodeos setup would include a baremetal server with 3.5Ghz or faster Intel Xeon CPU, and directly attached SSD or NVME, preferably of datacenter grade. 32GB RAM was quite sufficient to cover the needs. The node state would be resided on the SSD storage. If ZFS is used with lz4 compression, the SSD I/O load was pretty low, less than 20% on average.

Things have changed. Now a typical consumer-grade SSD drive would be almost permanently busy writing, and datacenter-grade NVME would be loaded at 30–50%. This would cause the node to delay behind the head block, or even unable to catch up completely.

nodeos daemon is using several subfolders in its data directory: /state is holding the current blockchain state in a file-mapped shared memory segment. This is where we need to focus.

/blocks is where the blockchain blocks are appended to the blocks log. The speed of updates us moderate, and a standard HDD is easily coping with this load. Also if state-history plugin is enabled, /state-history folder is populated with history data. This data is also gradually appended, so I/O performance of a spinning hard drive is not a problem (the only issue, you need to size the storage appropriately).

There are several possibilities to improve the state performance:

• Change the state database mapping from file-based shared memory to full in-memory mode: database-map-mode option is set to mapped by default, telling the Linux kernel to maintain the state memory in on-disk file (and that’s what causing the I/O load). Two other options will load the whole state into server RAM: database-map-mode=heap will use swappable server RAM, and database-map-mode=locked will reserve an unswappable RAM segment. In both cases, the server needs to have enough RAM to accomodate the state. In heap mode it will be able to use swap memory, but swap performance will not be sufficient for the node to operate. The benefit of these modes is that you have a copy of state on your storage, so in case of the server reboot, it will be able to start automatically, although from an old state. The drawback is that it loads the whole state on startup and unloads the whole of it on shutdown, so the node restart can take several minutes.
• Mount the /state subfolder to a tmpfs filesystem. This kind of filesystem is allowing you to store files directly in swappable RAM. Also the Linux kernel is smart enough to recognize that a shared memory segment is mapped to a tmpfs file, and will not keep another copy in memory. Nodeos read-write operations will be performed directly in tmpfs file. The node will be able to restart instantaneously. The drawback is that the filesystem will be empty after server reboot, so you will have to launch the node from a snapshot, or you need to automate the shutdown sequence to copy tmpfs contents to a nonvolatile storage.

If you use both tmpfs and ZFS filesystems on your server, and tmpfs is mounted to a folder in a ZFS filesystem, you need to tune the server boot sequence, so that tmpfs is mounted after all ZFS partitions are online. The following is an example of /etc/fstab file entry:

tmpfs /srv/waxapi01/data/state tmpfs rw,nodev,nosuid,size=48G,x-systemd.after=zfs-mount.service 0  0

Current WAX state is occupying 25 GB on tmpfs, and it has grown 30% in less than a month. So, you need to plan your servers accordingly. It doesn’t make sense to have less than 64GB RAM on a server, and 128GB is strongly advised.

With in-memory state, the server doesn’t need high-speed storage any longer. Even a virtualized storage is good enough, but renting a VPS with 128GB RAM is much more expensive than a baremetal machine with this much RAM and dedicated CPU power.

a 3.5GHz Xeon would still be good enough, but it makes sense to provision 3.8 or 4.0Ghz CPU if available.