Dear Community,
On March 7 at approximately 2:40 AM, our team observed unusual borrowing activity occurring on the Tender Protocol. A user had borrowed almost all available funds from the protocol’s pools. We saw immediately that deposits had not increased enough relative to how much borrows had increased, and we proceeded to check the sufficiency of the borrower’s collateral. Although the borrower had deposited just 1 GMX of collateral, the user was able to borrow $1.59 Million dollars worth of assets. Our hearts immediately dropped, and we felt sick to our stomachs- this was our nightmare.
It was 2:40 AM in Denver, and we were staying together in a small house. We struggled not to panic as we came to understand what had happened. We checked again, hoping it was a UI error- it was not…. We woke up the engineer asleep in his room. The Tender team gathered in a bedroom with our laptops and started to piece together what had happened as reality set in.
The previous day, at 4:56 PM, we deployed an upgraded price feed that pointed to a new oracle. This new price feed pulled the price of GMX from Chainlink rather than from a TWAP, as our developers agreed this should have been more reliable. This new code was written on February 1 and subsequently audited by PeckShield several weeks prior to being deployed on-chain.
While investigating the incident, we discovered that the code integrating the new oracle contained an error, and was returning a number with too many zeros behind it. This type of bug is notoriously common in Solidity contracts, which store numbers as integers without decimal points. Often, the decimal place is implicit, and the programmer must account for the precise number of decimals elsewhere. For example, the number 1,000,000 in Solidity could potentially denote 1, 100, or 100,000 depending on the number of decimal places applied.
A decimal error caused the contract to return the price of GMX with 38 decimal places instead of 18. This misstatement of the price of GMX allowed the borrower to use 1 GMX as sufficient collateral to borrow almost all available funds on Tender.fi. For a brief period of time, between 4:56 PM and 2:42 AM, the contract treated 1 GMX as having more value on Tender.fi than all Bitcoin in existence.
Between approximately 1:23 AM and 1:44 AM, the borrower was able to borrow almost all the available funds on the Tender Protocol.
With the intuition that our new price feed was most likely a vulnerability, we reverted to the previous price feed at 2:42 AM. At 2:57 AM, we paused borrowing for all markets to exercise an abundance of caution to protect our users. At 3:05 AM, we made an announcement no project wants to make:
Immediately after informing the community, we initiated attempts to reach out to the borrower. Thankfully, at 3:28 AM the borrower left an on-chain message:
It looks like your oracle was misconfigured. Contact me to sort this out.
At 4:19 AM, Coindesk published the story.
We responded with an on-chain message and also reached out with a private message over DeBank. We anxiously waited and obsessively refreshed the pages. By 4:20 AM, we had established communications. We proceeded to negotiate the repayment of all improper loans in exchange for the borrower keeping 62.158670296 ETH as a bounty.
We then sent the following on-chain message to the borrower’s address.
By 11:29 AM, the borrower had made good on their word by repaying all of the loans minus the 62.158670296 ETH. The Tender team was tired but relieved. We had just successfully resolved the protocol’s first and only security incident.
After a sleepless night, as the sun rose- we looked at each other in silence. This had just happened… We recapped our night together.
Since then, we have worked quickly and meticulously to investigate the incident and institute policies to prevent similar situations from happening again. To say we learned from our mistakes would be an understatement. We have come out stronger and more battle-hardened than ever before. While this experience is not something we would wish upon anyone- we are thankful we had each other to lean on.
How did this vulnerability happen?
Bugs happen, but a thorough and rigorous testing process typically prevents them from making it into production. Here’s how this bug made it past all our testing:
- When we tested this contract before deploying it using Hard Hat, we printed out the price of every token and verified it. Multiple engineers did a spot check on the prices, and it looked like the real price. Unfortunately, we may have accidentally tested a cached build artifact (a prior version of the compiled contract).
- The code with the bug in it was written 5 weeks before it was deployed and at the time we deployed it, we assumed it had been thoroughly tested already because it had passed its audit.
- We were working on launching the Claim button on the website at the time of the oracle deployment, which led to the UI remaining connected to the old oracle. If the UI had been updated on a branch and tested against the new oracle, we would have quickly seen the discrepancy.
- Our auditor didn’t catch this bug although this contract was covered by the audit.
Our next steps
- Deploy a new oracle contract, completely rewritten and deeply tested.
- New code: https://arbiscan.io/address/0xa11BAde71dF9005f4Cfb6FfeCd266eD8046Fd5c6#readContract
- Unpause borrowing
- Repay the unpaid debt left behind by the borrower.
- A plan of action will be addressed in an upcoming article.
Process changes we’re going to make
- Increase our highest Bug Bounty to $100K in Immunefi. So it is in everyone’s interest to go through our official Bug Bounty program.
- Every deployed smart contract will be manually tested on-chain before hooking it up to the rest of the protocol.
- Every deployed contract will be integration tested against the front-end on an unreleased branch, before going live.
- Running automated tests on our Continuous Integration host in addition to developer computers.
- Add an additional Audit Partner to our workflow
- Partnering with an active threat monitoring service so that we can more quickly identify suspicious behavior and contact the owners of relevant addresses. Furthermore, this will allow us to trace any exploited funds to exchanges should this happen again with a more nefarious actor.
- Emphasis on a stronger culture of testing: Internal standard operating procedures are being designed and implemented with guidance from our senior advisors.
We are grateful for all the love and support we received from our community during this incident. We wouldn’t be building this without you.
The Tender team building with friends in Denver hours before the incident.
Finally, the support we received from our community during this incident was truly incredible.
And my personal favorite
Our Roadmap
With this behind us, we are excited to get back to building an awesome product.
Here are some of the exciting things we’re working on:
- Growing our engineering team with open-source contributors
- Improving the UI for a better user experience
- Building leveraged and delta-neutral vaults for GLP to automate the looping process
- Talking to partners about building new strategies and collaborations
- Building toward full decentralized governance
- NFT collection to help reach a wider audience and offer special boosts to users
