Problem

Hubs (and our backend) don’t currently handle block re-orgs.

Current Behavior

Assume there are 2 blocks at the same height, 1a and 1b. Initially 1a is seen as the correct block and block 2 is created with 1a as the parent. As soon as block 2 is seen, hubs will create OnChainEvents for all transactions in block 1a and emit events which are then picked up by the backend.

If at a later point, 1a is re-orged away and 1b is determined to be the correct block, the hubs that have a latestBlockNumber > 1 will never see this change. This means:

• Any transactions that were in 1a and no longer in 1b will still remain as OnChainEvents in the hub database (and event consumers like the backend)
• Any transactions that were in 1b and not in 1a will not be picked up.
• Any transactions that were in both will be unaffected (but with incorrect metadata, which will lead to sync trie conflicts for the onchain events)

Hubs that missed the re-orged block (e.g. were down and started up after the reorg) will have the opposite (and correct) behavior of only considering block 1b.

The network state forks for all the fids/signers with affected transactions where some hubs will consider them valid and others will not. There is not current way to recover from this across the network without manually restarting hubs with —l2-clear-events which will delete and re-sync all onchain events (currently takes hours).

Solution

In the networkConfig file, add two new fields, a l2ReprocessNonce and l2ReprocessFromBlock . If a hub has already seen the nonce, it will do nothing. If the hub has not seen the nonce and the l2ReprocessFromBlock is non-zero, it will:

• Pause L2EventsProvider so it doesn’t process new blocks
• For each OnChainEvent in it’s DB with a blocknumber higher than the reprocess block
  • Delete the onchain event in the db
  • Emit a new hub event type HUB_EVENT_TYPE_REVOKE_ON_CHAIN_EVENT **with the deleted event
  • Enqueue a ValidateOrRevokeMessagesJob for that fid (bypass the recently changed signers optimization and validate all messages for all signers)
  • Clear the storageCache for that fid so it can be recalculated without the bad event
• Once this is complete for all affected block numbers, set HubState.lastL2Block to the l2ReprocessFromBlock. sync historical events from the re-process block and start unpause the contract event watchers.
• In order to do this, we’ll need a new secondary index for onchain events by block number (and associated db migration)