TL;DR: for a small network, modify the system contract and make sure to run at least 5 nodes. Always back up the keys!
EDIT 2023–05–28: it’s sufficient to have 4 producers to tolerate one BP outage.
EDIT 2025–02–13: it’s sufficient to run 7 producers to tolerate two BP outages.
A fully equipped EOSIO blockchain needs 21 active producers and a number of standbys for stable operation.
What if the project is small and running a few dozens of nodes becomes expensive? There are a few factors that need to be considered.
Servers sometimes fail, and sometimes they need to be taken from operation for software upgrades and system maintenance. In EOSIO Network, active block producers are listed in a schedule, a list of accounts that are supposed to sign blocks in a defined sequence. Each producer is given a window of 12 blocks to sign before the next producer starts its window. If a producer node is not ready or unavailable, there is nobody to produce the 12 blocks, so all speculative transactions are delayed until the next producer starts signing.
Once a block is signed, it is validated by other producers in the schedule, and goes into irreversible state after 2/3+1 producers have validated it. So if 1/3 or more of all producers are offline, the last irreversible block number would not increase, and the blockchain will effectively stop.
So, if we need to take a producer offline, unregprod action is needed to remove it from schedule. But line 100 in voting.cpp prevents the schedule from reducing. So if you have less than 21 active producers in the schedule, unregprod will have no effect. So, a small network needs to modify the system contract and remove this constraint from voting.cpp.
How big should the network be for stable functioning? With only 4 producers, the network will tolerate 1 producer going offline. If more than one are offline, LIB would stop moving and the network will stale. With 3 or less nodes, outage of one node will break the operation. With 7 producers, two nodes can go offline without breaking the network.
Also make sure that producing keys are backed up properly. If you lose them due to a system disaster, there’s a chance that the network would stop the operation forever.
