About 2 months ago, I was leaving a restaurant in LA after dinner when I noticed that my cell phone had no signal, so I restarted the phone (which normally fixed it) and was greeted with the following text message.

I immediately knew someone was either in the process of breaking or have already broken into one of my accounts, so I quickly sprinted across the parking lot, jumped in my jeep, dialed T-Mobile, and slammed on the gas. Somehow, I already had cell service back on my phone. In a panicked call as I was careening down the 405, there was some confusion between a pin and a password with the support rep and they kept asking for my password. It wasn’t until later that I find out they should have asked for my pin, I was never going to give someone my password and you shouldn’t either. T-Mobile accounts also have a PIN which is the number you give them as a private key. This was the first time I ever needed support and wasn’t familiar and the support rep wasn’t explaining so the call ended with them just recommending I visit a store in person the next day, because it was 9:30pm. Thanks for nothing.

When I got home 15 minutes later, I quickly checked my phone and saw the damage.

They got me. The bastards got me, Clint.

It’s weird, seeing one of your accounts compromised like that. There’s a first time for everything I guess. Coinbase? I barely even remember having an account on there, must have been at least 5 years old. I didn’t have anything in the accounts last I remember, that’s why it went dormant, but then I remember that my checking account is linked to it.

As I approached my front door, I recalled seeing something in the news regarding malware and Coinbase so I quickly googled ‘Coinbase sim hijack’ and was greeted with this headline.

Oh…Damn.

If it’s embarrassing for a crypto engineer, how embarrassing is it for someone in security? Really, this is a very embarrassing story for me to admit to, but there is a lesson in here so let’s keep going and try to have a good time.

I practically kicked the door to my apartment like SWAT and ran to my desk and validated that yes, I was locked out of my email account. It was a little surreal to be honest, but I had no time to get all existential and shit, I had an active breach I needed to resolve.

Currently, I was in step 3 of the Incident Response Life cycle, Containment, Eradication, & Recovery. Which also works as a sick band name.

In that moment, I didn’t know the full extent of the compromised accounts or even how they got in. I quickly saw news results of malware sniffing some password manager passwords and to be safe, I started kicking off virus scans of everything, I reset my main password and some of the bigger ones connected to my finances because I wasn’t sure and wanted to be safe.

My phone had service again because T-mobile was able to detect the SIM change as fraud and had already swapped my number back before I got in my Jeep, so I was able to utilize the 2FA SMS settings on my Yahoo account to log back into my email and change my account password back and then try and login to Coinbase, which was an an account I experimented with, but haven’t used in years. Eerily mirroring this story I saw of a guy who got $25k taken out of his checking account with the line, “Coinbase account I experimented with in early 2018 that was never closed”. I didn’t have my Coinbase credentials in my password manager because the account went dormant before I even started utilizing a password manager, so it was an unknown risk. But it sure as hell wasn’t unknown anymore.

I reset my password on Coinbase and I successfully login to see this.

Either the attackers just turned on 2FA to further add another hoop I have to jump through or I had an old 2 Factor account I don’t remember. So I then file a claim with Coinbase and have to send a photo of myself giving my webcam a resting bitch face and my ID to recover the account and it might take up to 48 hours so here’s to a nice relaxing 48 hours.

I called my bank and alerted them and they locked down what they could, but at the time, I wasn’t sure if it was still too late to recover any money taken from my account. At the very least, the 2 accounts compromised were recovered. It’s 2am, I should probably get some sleep, but I had to keep digging. I went through old emails and checked names of old authenticator apps and found that I did actually have an old account to one. The adrenaline high I was on didn’t allow me to go to sleep until about 5am.

After a nice “relaxing day filled with zero anxiety whatsoever”, I finally received an email in the evening informing me that my Coinbase account is recovered and I login to see this.

Oh neat, talk about the least chill way to find 50 bucks.

So it appears that back in 2013, when I setup my account, I did so with non SMS based 2 Factor Authentication. That was the control that prevented the attacker from logging into my account in the end. If I would have felt lazy that day and said I would do it later and never did, they would have been able to withdraw essentially all the money I had in my PNC Checking and Savings account. But now instead, I actually discovered money I thought I didn’t have and thus made money from this breach? That’s not exactly driving home the lesson I’m trying to give here though. I could have lost practically everything.

How Did This Happen?

I first looked into T-mobile since the issue started with the phone, I saw a breach very recently in the news and thought they may have swapped the SIM with my account creds until I realized that the T-mobile account isn’t associated with that yahoo address at all so that means that even if they did breach that T mobile account first, they never would’ve know to use it with my yahoo email.

I know what you’re probably thinking, Logan, why the hell do you use Yahoo Mail? Especially since the FBI or NSA had a backdoor years ago? It’s a free email account that I can use for anything spammy or craigslisty. I moved everything important to a private domain email years ago, or at least I thought I did.

Like Gandalf hauling ass to Minus Tirith in the beginning of Fellowship, I quickly ventured to HaveIBeenPwned.com to check my email and bingo.

This yahoo account was in a couple of different breaches before I moved to aliases and I’ve changed my password numerous times, but this is the first time it was in a breach in combination with my phone number. That reminded me, so I went digging and about week earlier I received this email late on March 1st.

At the time, I quickly changed my password to an extra long hashed password this time and ensured that I had some sort of 2 factor enabled, but there was a flaw. Despite those 2 factor settings, my Yahoo settings allowed me to recover my account via SMS, and that made me vulnerable to a SIM hijack. Adding to it, using the same SMS number for 2 factor authentication nullified the entire point of having a separate 2nd factor in this case.

At the end of the day the accountability lies with myself, but at the same time, I feel like Yahoo should warn users if those settings are selected. To show just how quick it can be, if I click on “Forgot password” with my email in the text box, I’m asked if I have my phone and it shows my partial phone number. I just have to confirm with a single button and then I get a text message.

Entering the 6 letters correctly like a Zelda puzzle is all someone needed to do to change my password and walk right into my yahoo account.

So if I want to login to Yahoo, I’ll need my email, password, and my phone. But with Yahoo’s SMS Recovery feature from hell, if an attacker wants to login, they just need my email and phone number to swap the sim

If I login to Yahoo traditionally with a password from a new device, I’ll get an email indicating that, but since I only was informed of changed password, I assume this is the method that was utilized.

When I finally talked to another T-mobile support representative later, I asked if they could see the history and what the previous activity was. They said it looks like the sim hijack was done what they said ‘through the system the stores are connected to’, but they didn’t say where the location was, but seemed as confused as if it wasn’t something they normally saw.

The problem with my thinking about the risk of a SIM hijack was generalizing the probability within the risk. My thinking was since I wasn’t a celebrity and I’m definitely not high enough on the totem pole of being a whale, the probability has to be what, 1 in n where n is the size of T-mobile customer base? That type of assumption is wrong because it assumes that there is only going to be 1 SIM hijack for the entire T-Mobile customer base.

You cannot properly calculate risk without context.

I don’t know the exact method the attackers used, but if I were to imagine someone with the ability of changing a sim having access to a list of Coinbase usernames, all you would need to do filter by ‘@yahoo.com’. Then from that list, try logging to every single one of them to see which ones are vulnerable to a SMS recovery.

You could easily automate this and within minutes probably have a list of Coinbase accounts associated to a vulnerable Yahoo account with their phone numbers. All that would be left is swapping the sim.

The regular routine of password changes I made to my Yahoo account did not resolve the overall vulnerability that persisted due to the configuration of both my Yahoo and T-Mobile accounts. That combined with an ocean of various attackers farming through multiple lists of emails, it seems more like it was actually inevitable and only a matter of time. This is why probability should be viewed less as a single static generalized number and more as one that increases over time without treatment.

My saving grace was utilizing proper 2 factor authentication with Coinbase. In contrast to how Yahoo operated in this case, Coinbase allowed a password change, but not a login without the 2nd factor. They could have changed my password a hundred times, but it would’ve mattered without the 2nd factor which wasn’t connected to my phone number.

Coinbase has a pretty nice account activity page and you can see from the bottom up, the activity leading up to my password being changed starting about a month ago.

This screenshot was taken 5 days after the compromise. My activity is removed. You can see there were numerous attempts on my account via the api, a couple days later was when I had that attempt into my Yahoo account, I believe that may have been to check to see if my account was setup for SMS recovery. It took about 5 days to finally swap the SIM card in the end, like this is a weekend gig for this piece of shit.

Lessons Learned

In the post incident phase, you need to take a moment to document the exact things you did wrong or didn’t do, things that you definitely need to fix in the future, and the things you did right, things that you want to increase to other areas if possible. First off, what were the things I did wrong?

1.) Not having a proper inventory of my accounts

You cannot protect protect what you don’t know. I did not have a proper inventory of all of my accounts in a single place until I got a password manager, but even when I did, I didn’t properly go back all the way into my email and find more dormant accounts. Especially accounts that are connected to my checking account. What the hell, Logan?

Also, consider this your daily reminder to get a password manager if you don’t have one. They also can also help assist you with creating that inventory of accounts.

Proper inventory in my opinion includes regular checks for breaches. Even though I have checked HaveIBeenPwned.com before, I haven’t checked in the last year or so when my phone number included in a breach with my email. Being aware of that might have been something that would have pushed me to remediate the next down the line issue.

One more thing about proper inventory is proper decommissioning, if I know I’m not going to use something, I should remember to delete the account rather than just delete the app.

2.) Not validating I was secure from SIM Hijacks

This was one I am really am eating it on. I knew of SIM hijacks, I’ve read stories on them, but I just kept kicking the can down the road for myself when it came to ensuring I was secure. I was always too busy. If you’re reading this and you haven’t validated that your SIM is safe with the instructions below I heavily suggest you do so.

For AT&T

AT&T allows a passcode to be added the account which would be prompted in situations like sim change. Their guide is here.

For Verizon

Go to VZW.com/PIN to set your PIN. As an alternative, you can stop by any Verizon store with your government-issued ID or call 1-800-922-0204.

For T-Mobile

To do this, you’ll need to call 611 from your T-Mobile phone or dial 1-800-937-8997 from any phone. The T-Mobile customer care representative will ask you to create a 6-to-15-digit passcode (or called 2nd PIN) that will be added to your account. I’ve also heard there is a way you can request that any bigger changes such as SIMs be made only in person at stores. However, it still leaves it up to T-Mobile to follow that.

For Sprint

Sprint requires its customers to have a PIN. It’s a good idea to periodically change yours by logging into your account. Once you’re in, select “My Sprint” and then “Profile,” followed by “Security.” Scroll down to “Security Information” to update your PIN.

For other carriers, just google them and sim hijack to find your solution.

I have heard of some people’s preferred solution was to get a Google Voice number for all of your SMS 2 factors, but I’m not really sure about tying more of my life into Google, so it’s up to everyone’s comfort level.

3.) Using SMS 2 Factor

Some may view this as steep, because some websites only give the option for SMS 2 Factor, but even if you follow the hardening instructions in the previous section, some articles have documented people using T-Mobile experiencing a sim hijack despite making the call to their carrier. So I don’t want to say that the call is meaningless, you should absolutely make the call if you haven’t but not knowing all the intricacies of the entire wireless system and how the system processes a SIM change request at a much deeper technical level, I would say it might not be 100%.

Researchers from Princeton University analyzed five prepaid wireless carriers’ authentication procedures to understand how they processed requests to change SIM cards. They signed up for around 50 lines across AT&T, T-Mobile, TracFone, US Mobile, and Verizon and then called to request a SIM swap on each account.

“We found that all five carriers used insecure authentication challenges that could be easily subverted by attackers,” they said. “We also found that attackers generally only needed to target the most vulnerable authentication challenges, because the rest could be bypassed.”

Danny Bradbury has a more in depth summary here and the full report is here. At the end of the day you should know that as of 2020, wireless carriers in the US aren’t treating this as seriously as they should. SMS 2 Factor shouldn’t be used unless where there is no other option. NIST depreciated SMS as a 2 Factor method in 2016, however using it as a 2nd factor is still better than not using any other 2nd factor at all.

So what did I do right?

1.) Using non SMS 2FA

I’m really beating a dead horse here, but if I didn’t use that separate 2nd factor for Coinbase, or if I would have used SMS 2 factor, they would have just gotten another text message and would have walked right in.

If you’re not sure of what to use, some password managers come included with an authenticator. Here’s a list I googled and am not affiliated with at all.

This was such a scare, I’m lucky my experience wasn’t similar to the horror stories I’ve seen online. When I played college football, I played under Joe Walton and he used to say that “winning covers a multitude of sins”. He meant it that you could have a really shitty game, but at the end of the day if you won, it didn’t really matter as much. I’d say maybe in this context, the right control can sometimes cover a multitude of sins as well.