Black Thursday at DeFi Saver

On Thursday, March 12 2019, the Ethereum market experienced its worst day in history, crashing from $194.97 at 1am to $136 (-30%) in the next 12h and eventually finding bottom at $95.87 (-51%) some 26h later. A quick recovery to $130 followed from there, but the damage had already been done.

As ETH price plummeted, it caused a cascading effect in the world of decentralized finance, but most notably in MakerDAO, by far the most popular DeFi protocol, but also one with the largest liquidation penalties.

Prior to the crash, our CDP Automation system managed 280 unique MakerDAO positions with over 77k ETH collateral within them. Some of these funds were unfortunately not successfully protected from liquidation throughout the crash and we wanted to share details on the circumstances this happened.

Network congestion

Although network congestion caused great problems for Automation when it was initially introduced, this was not the case this time. The system adjusted correctly and automated Repay transactions were going through without any issues. For most of the day, this meant running transactions at 200 Gwei, which resulted in over 300 ETH burned for gas that day once our reserves of GasToken depleted.

As a reminder, Automation users are charged for a max of 40 Gwei per transaction, with everything above that covered by us. This is a hardcoded limit in Automation.

Price updates in MakerDAO MCD

The way price updates work in Multi-collateral Dai is by taking a median value from those provided by multiple approved oracles and scheduling this value to be fed into the protocol as the current price after a 1h delay. This is something we wrote more about here.

There are good and bad sides to this in terms of Automation. The good side is that a price update leaves a full-hour window for all needed adjustments to go through. Sometimes multiple Repay transactions are needed to get the user’s position to the configured minimum after a price update, so having such a long window is helpful.

The bad side is that any price update could be so drastic that it pushes a user’s position from above configured Automation minimum to below 150% instantaneously. This is unfortunately exactly what happened for many users on Thursday.

At noon, a price update in MakerDAO was scheduled from $168 to $166 in one hour, at a time when live prices on most exchanges were already below $140. What followed next was a two-hour pause in price updates, most likely due to network congestion and unexpected gas prices which were already well above 100 Gwei for anything reasonably fast. At 2:12pm next price update was queued to go live in one hour at $136.

This move from $166 to $136 was about to push 80 out of 280 CDPs from above configured minimums (Repay below set at 170% to 175%) to under 150%. A price update that would completely skip the Automation Repay window for many users was inbound.

At this point we decided to manually pay back users debt to keep them >150% after the update goes through wherever possible and eventually got this number down to 30 after spending near to 6k DAI, with some positions being simply too large for us to cover. However, because of the additional delay on liquidations that followed once the price update went through, more users successfully got back to a safe ratio before going under. We believe the final number of liquidated positions that were in Automation to be around 20, though it’s difficult to be certain because of many users enabling, disabling or updating their settings as the day developed.

We would like to clarify this wasn’t a consequence of a design fault, but rather a design choice. Automation is currently a fully trustless system, which means that it only works with data that is on the Ethereum blockchain. As of now, we only have access to the current price in MakerDAO on-chain and therefore there is no way for Automation to act upon next price changes until they become live. We have requested access to the next price data on chain from Maker multiple times, but to no avail. Trustlessness of the system is something we pride ourselves in, but we believe the choice to step outside of it will be a better option for the time being — more on this later, with information on other Automation updates.

Users losing 100% collateral

At this point of the day, another unexpected development started happening — bids of $0 were going through for liquidated collateral. This meant that some users were losing 100% of the collateral in their positions, instead of the expected (slightly higher than) 13% liquidation penalty. While this wasn’t in any way connected to DeFi Saver or Automation, we still feel it’s an important issue that also needs to be highlighted as many users were affected.

There is as of now no way for anyone to take part in these auctions, except to run a Keeper — a bot account configured to bid funds that it has access to. However, due to the size of market movements on Thursday, current Keepers were too few to keep up and eventually ran out of funds. Once this happened, someone with enough knowledge of the protocol jumped in, bidding $0 and winning a great number of auctions. Unfortunately, this resulted in over 5m DAI of bad, undercollateralized debt which will be recapitalized in an auction on Thursday, March 19.

In order to help alleviate this, we will soon release a simple interface for users to be able to take part in collateral auctions. In theory, one could even win an auction on their own collateral. We believe that keeping the MakerDAO protocol as healthy as possible is in everyone’s best interest and making the auctions process more accessible should reduce the liquidation losses to a minimum.

Plans for Automation updates moving forward

Our plans for Automation updates haven’t really changed much in the light of these events, but Black Thursday has highlighted how urgent they are. Below are the most important changes we plan to introduce and that we aim to have live in Automation within the next few weeks.

1 —Extended Repay and Boost powered by flash loans

We have recently introduced multiple flash loan powered features, including Extended Repay and Boost. However, the important thing to note is that these are currently only available for manual utilization in our MakerDAO dashboard. Automation is reserved for functions that have been heavily tested, which we can now say is true for these, due to the sheer amount of volume these features have seen over the past week.

Including these into Automation will greatly reduce the number of transactions Automation needs to send out, while allowing for greater volume at the same time. This is because the first step of these extended features is a DAI flash loan that pays off the needed part of debt, allowing for much more play room. Ultimately, Extended Repay could actually be used even with the ratio under 150%, which would have potentially saved all automated positions on Thursday, given how delayed liquidations were.

As a reminder, the first step in traditional Repay is taking out required ETH from the CDP to use for obtaining DAI for debt payback, meaning that it’s a method hard limited by the minimum 150% ratio and cannot be applied when below that. Additionally, for any position where automated Repay needed to be applied after reaching, for example, a ratio of 154%, the system might need more than 5 transactions to get you to the configured minimum of 170%.

2 — Injecting next price into Automation

Ever since the introduction of Multi-collateral Dai and the updated price feeds, we had plans to add support for the next price value, so that the system can react upon it where needed, even before the price update actually goes through.

As of now, we could not obtain smart contract access to this data from Maker and it does not seem likely that this will be provided publically or shared with us privately anytime soon.

Because of this, we plan to introduce a method for us to approve a next price update as a value that Automation can act upon, but only in cases where this update would push a position into liquidation. This method will be replaced with on-chain next price data as soon as it becomes available to us. While we value trustlessness greatly, this is something that has already been suggested by our users recently and we believe it to be the right choice for now. If you have any feedback regarding this, please let us know.

Once these changes go live, a manual update to the new version of the system will be required from users, so please keep an eye on our Twitter or Discord.

We will be sharing more details as soon as the update is ready. Until then, we recommend all Automation users a minimum 200% ratio threshold for Repay to allow enough room for any large price updates.

Stay safe out there.

Stay connected:
🌍: DeFiSaver.com
💬: Official Discord
📢: Official Twitter