Automatic CDP protection faux pas — analysis, updates in place and next steps

We introduced the CDP Automation feature three weeks ago, offering users the option to enable automatic liquidation protection, as well as automatic leverage increase depending on ETH price movements for their collateralized debt positions. This was the crowning of our continued efforts to provide users of the Maker protocol with advanced features for CDP management and, ultimately, a means to protect their positions from liquidation at any point in time.

A quick introduction to CDP Automation

DeFi Saver was first introduced with Boost and Repay features available for manual use for all CDP owners. These quickly became fan favourites and we were thrilled to continue working on automated management with the feedback received from the community. We created a trustless system where the CDP owner provides rights to execute Boost and Repay to a smart contract with clearly defined rules that cannot be bypassed. Neither Boost or Repay can be executed while the CDP’s ratio is between the minimum and maximum configured ratios. The owner can change these rules or remove the provided privileges by turning off Automation at any point in time.

In the current, beta iteration, we purposefully limited the system to only be accessible to our bot and to only allow it to make required CDP adjustments. The goal is to keep the added risk for using Automation as low as possible and to prevent any potential attacks. The bot monitors CDPs subscribed to Automation and executes Boosts and Repays when the parametres allow and require these adjustments. The bot’s gas expenses are covered by managed CDPs, but this is also limited (max gas price paid by users is 40 gwei, with anything over that covered by us) so there is no way for the bot to drain any CDP.

The feature was introduced as BETA, to highlight something could go wrong. And went wrong it did.

What happened

On September 24, the price of Ether went plummeting, going down 8% in just 30 minutes and almost 15% over the course of an hour and a half. At 8:52pm (CEST) we first noticed that some of our Repay transactions got stuck.

Although we always understood how volatile the market can be, what caught us off guard was the spike in gas prices that happened in a very short period of time. Our Automation bot was always designed to use high gas price straight away, by taking the fast preset from ETH Gas Station and adding a few more gwei on top of that depending on conditions. This worked perfectly and we haven’t experienced any delayed transactions up until that point. It was actually our design decision to keep system complexity low and not include a re-sending mechanism during this phase, even though it was a planned addition and one we were already working on.

Once one of our bot’s Repay transactions got delayed, what followed was a cascading effect for all others. The gas prices for these transactions were simply too low to cut through other pending transactions on the network that was already congested prior to the price drop. Our gas price estimate system simply did not adjust quickly enough. At this point the system had alerted us about the issue, but in an unfortunate turn of events this was around 9pm locally, a time when most of our team members are away from keyboards, which additionally prolonged the delay. In the next hour we pushed multiple system re-configurations in attempts to kick-start and drive our Repay transactions through. Something that finally happened as we reached a gas price of 45 gwei.

After the initial wave of delayed transactions went through, the system quickly stabilized and proceeded to work as intended. Unfortunately, the whole event did not end without casualties and a few CDPs were liquidated in the process.

Immediate updates

Once things were stable, the first thing we did was come out to Twitter to provide as much information on what happened as we could and own up to the issue. We never offered any guarantees and expect our users to be aware of the risks involved, but we nonetheless offered to compensate for the losses suffered by the owners of the liquidated CDPs. We hold ourselves to the highest standards and we wanted to offset these loses, as the community of people collected around DeFi Saver has been nothing but supportive from our very beginnings.

Secondly, our developers immediately went on to finalize a basic version of the gas resending system for deployment and, after verifying it behaves as intended, it was integrated late the next day (Sep 25). This system tracks all pending transactions and re-sends with 1.5x the previous gas price in case they get delayed. The theoretical outcome of applying this, currently live, system to the network circumstances experienced on September 24 would be effective protection of all CDPs with no difficulties.

This is currently a rudimentary system and we additionally plan to account for multiple variables when deciding how soon transactions get resent and with how much increased gas prices. These variables will include how quickly the value of Ether and gas prices change, as well as how close a CDP is to liquidation.

Furthermore, we have also implemented better admin tooling that will enable us to respond quicker to any issues with the system in case they were to happen again. And finally, something that our users have perhaps already noticed, the DeFi Saver’s frontend now features a notifications system that will allow us to target any users with important warnings or updates.

The future of Automation

As we mentioned previously, the current iteration of the whole system is an early, beta version. We value decentralization greatly and the ultimate goal has always been to create a system where any entities can run their own bots and take part in protecting or leveraging users’ CDPs. We are considering a version 2 of the protocol to be open to the community, allowing other bots to take part in order to increase security and make the system more robust. This would include standardizing the system in such a way that other defi apps can easily integrate Automation, too.

In the meantime, we are also looking into options to flash-lend funds to CDPs nearing liquidation as an even quicker way to rescue them from liquidation. And we should mention that we are already working on Automation support for MCD, as well as Compound assets.

We thank you for your understanding and the support received over the past few days and, really, since the very start. We are also humbled by the influx of new Automation users and are very proud to share that the total value of the collateral of CDPs managed by the system has passed $1,350,000 today.

As always, feel free to reach out with any questions or feedback in the comments, via Twitter or in our Discord.

Stay connected:
🌍: DeFiSaver.com
💬: Official Discord
📢: Official Twitter account