Keeper Infrastructure Improvements

One of the biggest technical challanges we face at RoboVault is developing and maintaining reliability and effective keeper infrastructure. The keeper is responsible for getting rebalaceDebt and rebalaceCollateral transactions on to the chain as fast as possible when they’re needed. These transactions help minimise the impact IL aiming to hedge against price movements & also ensure that the vaults do not experience liquidations by keeping the LTV ratio within a specific range.

Last weekend, December the 4th, there was a “flashcrash” within the entire crypto market including on Fantom, with a 40%+ drop in price over a few hours. History tells us we should expect events like this every couple of months. Based on past performance we thought our keeper infrastructure would be able to handle the high volatility and network congestion. Unfortunately, a few minor bugs led to losses in our WFTM vaults

We have a maximum gas price set to avoid burning FTM on gas, however, there was a significant gas price spike on the network well beyond our limit, as a result, the collateral rebalance transactions were delayed by 17 minutes. All our vaults hold positions on Scream, during these 17 minutes our 3 WFTM vaults were liquidated. The damage:

WFTM Multistrat -2.4%

WFTM B -4.2%

WFTM C -1.3%

A total loss of ~$39,500 USD across the vaults. We will be reimbursing all users of the losses over the coming months.

We’ve identified 3 key issues that need to be resolved, single point of failure, effective gas pricing and fast broadcasting.

Historically we have run a single keeper for each strategy, this means if this one keeper fails during congestion, the vaults risk liquidation. We need redundancy. This past week we have been working on leveraging Gelato and Open Zeppelin Defender as keepers for vaults. The Gelato tasks are live and the Defender Keeper will go live in the coming week. These three keepers will always be competing — let the best keeper win!

We have written a custom gas station that calculates the median gas price of the previous block, which we then multiply by 1.5 to make the transaction competitive. Our data suggests this approach is working well, however, there is clearly a need to increase the maximum gas price. This we have done, and we have also deployed logging so we can monitor the gas prices during congestion and increase the limit further if needed.

The Gelato team has informed us they have no gas price limit meaning the Gelato keepers should reliably be able to complete required timely transactions during network congestion.

Although the Gelato and the OZ Defender keepers will likely meet our reliability and performance requirements, we will still continue to improve our custom keeper infrastructure. One improvement we’ll be making is multicasting to multiple node endpoints, this will ensure our transactions land on a healthy number of mem pools.

Related posts:

In this article I will detail a possible approach using Resource Tokens and Azure functions. Let’s go! The recommended way to use resource tokens by Microsoft is to setup a Mid-tier service, the…

During the Christmas holidays, I expect many of us will be attending some in-person parties with guests we do not know very well e.g. friends of friends or a distant relative we have not met before…

Karen Adams loved to gamble. The 54-year-old worked two jobs and didn’t have a lot of free time, but she enjoyed going to the Mountaineer Gaming Resort in New Cumberland, West Virginia, when she had…