Greetings all! Massive apologies for the delay getting this out.
TL;DR - Governance proposal #6 failed due to a bug introduced (by me :shame:) as part of the Minter component upgrade/migration that was done in Proposal #4.
Background
Gov proposal #4 changed the way that liquidations work in the Kolibri protocol by creating a new minter component, then swapping it into place. This is the first major component upgrade that’s happened to the protocol, and thankfully things seem to have gone off without a hitch.
Kolibri.js is a library we’ve built to encompass functionality used by the main kolibri app and the governance portal, as well as some other bots/scripts/etc running in the ecosystem. It’s also considered the de-facto source of truth with regards to contract addresses.
And lastly, as a reminder the entire system currently has “break glass” contracts sitting in between the DAO and individual protocol components, forwarding requests from the DAO to the proper component, but allowing a multisig to circuit-break any individual components were a security issue or some other bug to be discovered.
The Bug
When we carried out the migration, I took on the task of updating the contracts in kolibri.js to point to the new minter contract that was deployed and adopted on-chain. This was a fairly standard change, but I forgot to update the corresponding break glass contract as well, meaning that kolibri.js 1.15.0 had a proper MINTER, but not the corresponding BREAK_GLASS_CONTRACTS.MINTER.
When filling out the governance proposal wizard on the gov portal, we use these values from kolibri.js to stitch into the lambda that’s created. Since the break glass contract wasn’t updated properly it lead to a lambda that was impossible to execute.
There are a lot of checks in the Kolibri components that have to do with authorization, so while the DAO could without issue submit a request to the break glass that was pointing to the old minter, the new minter won’t respond to the old minter’s break glass contract (we decided to isolate these components and just consider break glass contracts as part of the underlying component to keep the system easier to reason about).
The Fix
The fix was super simple, it was just a single line change that restored proper functionality by updating the break glass contract to the proper place.
Mitigations & Future Work
There are a few things that could’ve caught this bug that we’ll be working on/implementing.
-
Sanity checks for the DAO proposal wizard:
It’s trivial to check as part of lambda formation that the break glass contract that we’re using is wired properly to the underlying component that it’ll be talking to. This check would’ve prevented this issue from occurring outright, and we can squash this specific bug from happening again (though there are similar bugs that are harder to sanity check). -
A #check-my-lambda channel on Discord
While we should have good confidence that the lambda generation wizard works as expected, some human eyeballs never hurts just to sanity check things before they’re submitted on-chain. An ounce of prevention is worth a pound of cure and all that, and we (hover labs) are already very active in the Discord, so we’ll be happy to spend some cycles sanity checking proposals.
As always, if anyone has any ideas on improvements that can be made to this process beyond what we’ve outlined please reply here so we can discuss!