174: Pacific Rim

Introduction

0:00Hi, I'm Jack Reesider, host of the show. Back in 2018, an interesting cyber attack took place. It's kind of a funny thing. I mean, it basically came onto my radar the second month I was working at Sophos. Oh, I should introduce you to Andrew. Yeah, so I'm Andrew Brandt, and throughout the time the research was going on for this story, I was a principal researcher for Sophos, but I am now a principal threat researcher for a company called Netcraft.

Researching Threats

0:32So one of the things Sophos wanted Andrew to do was research novel threats and write about them on their newly established Sophos blog. The team that I was on eventually didn't exist. I was the only person on it, and one of the analysts reached out to me through the company chat and said, Hey, I've got a great story for some really cool research. I'd like to write it up and have you publish it on the blog and do some edits on it. I said, great. Tell me more. And he told me the story, but the one thing he didn't tell or what he said he couldn't tell me was who the target was.

1:09So he's like, okay, fine. Send me what you got. Let me research it, and I'll write about it.

The TV Set Incident

1:13It started with a TV set. So there was a sales office, and they had a bullpen like you have in a lot of sales offices where people are on the phone trying to sell the product. And so they had this leaderboard that was on a computer screen that was running off a little Linux computer. And that was the first machine that got infected. And the threat actors managed to pivot from that Intel Nook, which is like a tiny little computer that's small enough it can mount on the back of a TV monitor that's hanging on the wall.

1:47That they were able to pivot from the Nook and find access to the repository where the source code was and then get into that. And then to do the cloud snooper attack on that cloud service where the source code was, it's just mind-boggling to me. Like the amount of effort involved in pivoting from this to this to this to get into this, and then to build this like backdoor that allows them access. It's amazing to me. Oof. The attackers got access to the source code.

2:19But why? Was this an insider trying to seek revenge? Were they stealing it in hopes to sell it to someone? Did they steal it so that they could copy the product and steal their intellectual property? At the time, nobody knew what their motive was.

2:38These are true stories from the dark side of the internet.

Sponsored by ThreatLocker

2:41This episode is sponsored by ThreatLocker. If you've listened to Darknet Diaries for a while, you've already heard of ThreatLocker. I've talked about how they lock environments down, deny by default, zero trust, all of it.

3:14But the problem they were solving changed because attackers changed. They don't break in like they used to. Now they just log in with real credentials, real sessions, nothing that looks out of place. Once they're in, they're treated like they belong. So ThreatLocker took what they already were doing and pushed it further with their zero trust network access and zero trust cloud access. So now access isn't just about logging in. It's about the device, the connection, and whether any of it should be trusted at all. If you want to see what zero trust looks like when it's done right, go to ThreatLocker.com slash Darknet.

3:49That's ThreatLocker.com slash Darknet.

Sponsored by Meter

3:52This episode is sponsored by Meter, the company building networks from the ground up. If you employ and work with IT engineers, you're going to know how hard it is for them to do their job well. What your business needs is performant, reliable, secure networking infrastructure. But what you get is IT resource constraints, unpredictable pricing, and fragmented tools. What you and your engineers need is a modern platform you can all trust to support your business. Enter Meter. Meter delivers a complete networking stack, wired, wireless, and cellular, in one solution that's built for performance and scale.

4:29Alongside their partners, Meter designs the hardware, writes the firmware, builds the software, manages deployment, and runs support. That means less time your employees spend writing to multiple vendors and more time working and improving your IT systems. Meter's full-stack solution covers everything from first-site survey to ongoing support, giving you a single partner for all your connectivity needs. Thanks to Meter for sponsoring this show. Go to Meter.com slash Darknet to book a demo now. That's spelled M-E-T-E-R. Meter.com slash Darknet.

4:59And go book a demo.

Firewall Hack

5:03So hackers broke into a company and copied the source code for that product. So we managed InfoSec there for a while, and currently, too, it was the type of network that was, you know, in the process of being brought over to a kind of set standard. This is Craig. He helped clean up the intrusion. So my name is Craig Jones. I'm the Chief Security Officer of Ontinue. But several years ago, I was actually the Senior Director of Information Security inside Sophos. I mean, if you don't know Sophos, we're a UK-based cybersecurity provider that has everything from kind of EDR, MDR, and through into firewall products.

5:41And at the time, they had three different firewall products, one being Cybrome, the other one being a German-based firewall provider, and the new Sophos firewall product. So essentially, they were collapsing two products into one, and the new one being Sophos firewall. Yeah, Sophos' main product is their firewall. This is a network device that will act as a wall between a protected network and an unprotected one. Out of the box, nothing is allowed to pass. You have to tell it exactly what you want to allow through, because the point of a firewall is to stop unwanted traffic from coming into your network.

6:17And believe me, there's a lot of unwanted traffic that's always trying to get into our networks. And in 2014, they bought another company called CyberRome, which was also making an interesting security product. That product, you know, we were flattening that product to make it into something helps. You know, like CyberRome was very much purchased to be the development house for the new Sophos firewall product. You know, there's some super hot developers there. And it was this newly acquired CyberRome network, which was the victim of this attack.

6:47Like, someone had gotten into CyberRome and was looking for their source code and found it for one of their products, which Craig and his team had to go clean up that intrusion. There's some really cool stuff that those actors did. You know, there were several points when I sat down and thought, damn, these guys really know what they're doing. You know, I think for me, there was one where they'd actually attempted to intrude the network in several different ways, mostly at the same time. And what was really interesting about it is we could tell that there were two or three actors working together in different consoles.

7:22And one of the things they did, which is kind of funny, actually, was that they'd gotten hold of a secure shell key. And one had obviously copied it, and another person was trying to type in the password for it. And we could tell in the logs that they were mistyping the password, you know. And, you know, the person who'd obviously taken the key had obviously tried to relay it onto another person, and they were mistyping this thing. And it was kind of crazy. You immediately knew then that this wasn't just, like, a dude.

7:55You know, this was a serious operation. The attackers had really unique methods for getting in, not methods that were publicly known at the time. Super sneaky and crafty ways to get into a network. And they got in through multiple ways. And then when they got in, they were able to move laterally in really unique ways, too. So unique that the Sophos team had no idea that stuff was even possible. It was like exploiting bugs in the way AWS handles identity. One problem, though, is that they didn't have enough monitoring at first to know exactly what these hackers saw or took.

8:26They assumed because they got access to the repository with the source code that they took the source code, but they were unsure. So they had to enable a lot more logging and monitoring to fully eradicate them from the Cyber Roam network. Andrew wrote this attack up because it was so interesting and new and published it on the Sophos blog, but didn't say who the target was. Yeah. So, flash forward, two years go by. It's now 2020. You know, we now have the team up and running. I've got a couple of people working with me. We're publishing a few blogs a week.

8:57And I find out from internal people within the company that there's a security incident. And the security incident started with a tech support call where someone sent an email to their support technician and said, hey, my firewall is showing this URL in the user interface and I didn't put it there and I don't know why it's there. Hmm. It sounds like a minor problem at the surface. This firewall had a configuration which showed what IPs are allowed to access it and manage it and configure it.

9:30And a strange URL was showing up in that list of IPs. It didn't make any sense as to why it was there or why anyone would ever even put it there. So the Sophos has a firewall called the XG firewall. At this point, it was just called the XG firewall. And the firewall has its own operating system. It's running a version of Linux in it. It has a UI that's running on the front of it so that you can manage it. At the same time, someone outside of Sophos submitted a bug into Sophos for this same issue.

10:03I think it was April 21st. They had, well, we actually had an external bug bounty report as a SQLI injection. And what was kind of weird about it was, you know, I remember the user actually claiming to be from Australia, but they had a Chinese name, you know. Now at the time, we didn't have amazing telemetry from any of the Sophos firewalls. So we had kind of base telemetry, which gave you like, it was really designed for product managers to understand what features that, you know, users were using.

10:42So they understood where to put their kind of limited resource time into, right? So we had that, and we had a really good idea of like, you know, where all of the serial numbers for these devices sat and their IP addresses associated to. So it's always kind of interesting to correlate the IP with the intended location of the researcher. So we got a researcher's device, it's one that had never been turned on before, which was pretty suspicious, you know, it had never been registered. It was a serial number that had just come from a web trial of a VM.

11:17And we found the IP actually related back to Changdao in China. Okay, odd. Someone from China with a trial license of the Sophos firewall found this bug and reported it to Sophos? And Sophos did, in fact, pay the bug bounty for this. It was about $10,000, I think. Hmm. Someone got paid a pretty penny for reporting this bug to Sophos at almost the exact same time that they were seeing it being exploited by devices in the wild. Strange timing.

11:48We called it Asnarok. So the team investigated this bug further. It was present in the front-end web user interface of the firewall. To configure this firewall, you can use a browser and access it that way. Well, the web UI of this firewall had an SQL injection vulnerability in it. Basically, in one of the form fields of the firewall, like maybe the username field or something, an attacker could enter in some commands there, which would glitch out the user input handling mechanism of the firewall and allow the attacker to inject their own commands into the database of the firewall where the configuration sat.

12:21And this was a really bad bug for Sophos to discover. Their devices are supposed to be blocking hackers from getting into the network, yet it's the vulnerable device, which is allowing hackers into it? This is not good at all. And they found that essentially every firewall that was facing the public internet was affected by this bug. These firewalls weren't just vulnerable. They all had been hacked into, exploited. Someone probably scanned the whole internet looking for these particular Sophos firewalls and then ran some kind of automation script to go infect them all.

12:57So, we kind of worked out that there were a huge amount of devices affected. I think in the aimed FBI report that came out about this, I think they mentioned 80,000. It has a guess it was probably more, you know? Hot dog. 80,000 Sophos firewalls hacked into. But just because someone put a URL in place where it shouldn't be, that's not all that damaging just by itself. So, the team investigated what that URL did, and that's when they started to panic.

13:30The URL would trigger a Git request in order to update the Sophos firewall itself. But what was really weird about it is that it was a W-Git to a domain called SophosFirewallUpdate.com. And Sophos didn't own that domain. So, it tried to blend in like it was supposed to be there, and it fooled many of the people, even at Sophos, who just figured the update domains changed. But my goodness, this meant suddenly 80,000 firewalls were looking somewhere else for updates and not to Sophos?

14:02And it's kind of strange because we actually monitor all domain registrations. It's kind of part of our kind of core security, like, ops function. So, every single, like, cert that was registered, every domain that was registered, we kind of pop up and, you know, anything infringed on SophosIP, we attempt to pull back, you know. And it was one that had popped up, like, a little while ago, but nothing had kind of come of it, you know. But actually seeing this thing in operation was quite, like, quite jarring, you know. And I don't know if you fully understand what this means. If a malicious hacker is able to send your firewall software updates, then they can put in whatever they want.

14:38They can give themselves full access to the firewall, or they can log all traffic going through it. They can poke a hole in the firewall and let themselves right into your network. And then from there, they can just infect your whole network with ransomware. The thing that is supposed to block unwanted traffic is no longer blocking anything if the attacker wants it that way. Not only that, Sophos was worried that they had lost capability to update any of their firewalls properly. Yeah, so effectively, what they could do, I mean, the truth is anything, but what they really were after was system configuration and passwords.

15:12Now, I've always suspected that this was something that they expected to run quietly, for them to kind of pull that configuration, the passwords, quietly, and then for them to kind of delete any presence they ever had on those firewalls. And then for them to have a really easy and simple access campaign. Jeez, so the attackers took copies of the configurations from the firewalls and then passwords from it? This was a pretty darn scary event for the Sophos team to handle.

15:42So it was very much like an incredibly tense situation where we first up had to get a hold of one of these devices. You know, we set multiple teams up to work out what happened and to really do some in-depth incident response on this. We're incredibly lucky, you know, we had the entire arm of like Sophos Labs to help us kind of reverse engineer this stuff. Okay, step one, fix the bug that made these things vulnerable. And step two is get the bug fixed on as many firewalls as soon as possible.

16:16They were able to complete step one pretty quick, but step two was a little bit more tricky. If you buy a firewall, whether for your home or a large enterprise, typically you've got to update it yourself. Just like how you have to do your own software updates on your phone or computer. And Sophos firewalls are no different. The customers are the ones who have to issue updates for this thing. But to Sophos, this was too critical of a bug to try to tell 80,000 customers, go update your firewalls. Because I'm just guessing that like less than 50% of them would do it in the first month.

16:47There's just not enough time or it's not a high enough priority for them to fix it. So Sophos decided to do something they've never done before. They pushed out a hotfix to these firewalls. A hotfix is like a little software patch that can run in real time. They can live update all the firewalls remotely with these hotfixes. It doesn't require the firewall to reboot to be enabled. And they felt like they had analyzed the attack and figured out exactly how the threat actors were leveraging their access.

17:21And they closed those loopholes with the hotfix. This was the first time Sophos ever issued a hotfix to one of their customers' devices. Now, they had built the facility to do hotfixes and they had not really used them before this. So there had been no real reason to do it. But I think they had built in the capability to do these hotfixes, anticipating that there might be an opportunity to use it if there was something that was a real problem. And it was fortunate that they had rolled this out in the previous firmware update just before this attack had taken place.

17:59Yeah, I think this is a really big deal. Like, it makes me wonder if there's language in the small print of the terms of service that says Sophos reserves the right to make configuration changes to your firewall or update it whenever they want. I think that's what's important as well. It's like, this isn't something that's just kind of done. And it's not something that's done willy-nilly, you know. And you're right. I mean, it does feel kind of offensive, someone coming in and tampering with my stuff, you know. But effectively, it's written into the EULA, like the End User License Agreement.

18:33And candidly, you kind of need this. And I think that's where a lot of firewall providers actually fail, is the fact that they rely on end users to patch everything. And candidly, so many firewalls are just bought and they're never updated, you know. Gosh, I really don't know where I stand on this. I was a firewall admin for my previous employer for 10 years. Those Cisco firewalls were my babies. I knew everything about them. I would review every single change that ever took place on them.

19:04And I don't think I would like it if Cisco just decided to patch them one day, without my consent. Like, some were in hospitals that were mission critical and some hadn't been patched for years because they were so finicky. And any change to them would just make them wig out and crash. And when I had to update them, I wouldn't do them all at once in one big swoop. I'd do them one at a time and hold their hand and make sure that nothing broke after the upgrade and everything came back up as expected. So if a security vendor just slapped a hotfix on all my firewalls that I was in charge of, I would freak out.

19:35What? We did not get approval for this change. We aren't in a maintenance window. We don't even know what changes you made to the firewall or what's happening. How can you just come into our devices and make changes without us knowing? I would be upset. Like, I wonder, did the Sophos team get approval from their lawyers before issuing a hotfix to their customers like this? Is this even legal? Yeah, I mean, that's a great question. I was not privy to those discussions. But I'm sure there were discussions like that about, you know, what is our legal liability?

20:07What are we allowed to do and not do remotely on these devices? I believe, ultimately, the decision was made, and I'm not sure if there were lawyers consulted on this or not, but it made a lot of sense that the harm of allowing the firewalls to basically try to ransomware the inside of networks was probably greater than the risk of somebody complaining that, oh, you made a change to my firewall without telling me first. So they just went ahead and did it. Yeah, I mean, I think not only that, but it's like this idea that the vendor can come in and change my device in any way.

20:44It's not just like crash logs that are being sent to it. It's, wow, what else can you do? If you could put a hotfix in, can you see the password? Can you see the connections? Can you see, can you come in and do other work? Can you update to different firmware that has malware on it or something? Like, could you do things that, you know, and, you know, you start, your mind starts going, like, could you do things that the NSA wants you to do and go and spy on this customer or something like that, right? And so when you're a firewall admin, you're like, no, I have to make sure that this is no other person in the planet can access this

21:18but me and other people on my team. Because you can't risk some, like, a backdoor. It's basically a backdoor that you had. Yeah, that's entirely accurate. And you're not wrong. And these are devices that are typically placed in a position in the network where they act as the barrier between the outside and the inside worlds of the networks. And I recognize that that is a risk. However, and it is also worth noting, that this is exactly what the bad guys were doing at this moment.

21:54They were installing malware inside the firewall. So, how do you fix that? I could just imagine the headlines at this point. And just, I don't, I don't, my question is, did any bad news come out to be like, Sophos found vulnerable, tens of thousands of customers impacted, huge vulnerability, hacker has complete control over their firewalls patch immediately. Like, that could make the stock tumble.

22:24That could, you know, really hurt business. Yeah, I mean, it could. And that was one of the reasons that I was brought in, basically, on day zero of this happening. The company realized that they had a public, a potential public relations nightmare on their hands. And they needed to communicate as openly and as forthrightly as possible, everything that they knew and everything that they were doing to fix it. And, you know, credit goes to the people, you know, in leadership at the company who decided that, you know,

22:56possibly against the, you know, conventional wisdom at the time, that they were going to go public with everything we knew about this attack. It was not a common thing at that time. But, as I said, you know, I've worked for a long time doing this kind of, in this kind of role where I do investigations and then, you know, publish about them to the public to warn people about bad things that are happening on the internet. And it's been my experience that the more information that you get out, the better protected people are.

23:31And that being radically transparent benefits everyone. It helps the customers who are affected. It also warns the public that, like, hey, this is something that you need to be aware of in the future. And it might also put the threat actors on notice that, hey, we're watching you and we're taking action to stop you. As the Sophos team investigated this more, they learned that whoever did this attack had to have really in-depth knowledge of Sophos firewalls.

24:03Like, there's no way they should have discovered this bug unless they had access to the source code, which wasn't publicly available. And that's when the pieces started clicking into place. The part of this firewall that was vulnerable was code from the CyberRome firewall that was moved over to the Sophos firewall. And two years before this, as you know, there was an attack on CyberRome. And what server did the attackers get access to? The one with the source code for their firewall. So they started to think, holy crap, this is a very serious threat actor who's been attacking us for years.

24:39They spent tons of effort getting into CyberRome's network to steal the source code, only to study it for bugs and then launch a massive attack on our Sophos firewalls? Whoa, what do you even do with this information? To think your products are the target for a major cybersecurity campaign like this? This is starting to smell like a nation-state actor is behind this. Who else has that much time and resources? And what the heck was the deal with someone from China submitting this bug the exact same time that Sophos discovered this? Very strange.

25:10One of the things that we've been kind of working on, but even before this situation was, you know, pulling in our telemetry or firewall telemetry, the kind of basic telemetry I was talking about earlier into Splunk. And I remember talking to Mark, who was just this amazing Splunk engineer in my team.

25:30Like I said, well, can we go back on that data? Like, can we find out, like, when this first started? Because I couldn't quite work out the exact moment in time or the first firewall that was hit by this Asnarok attack. And then I went back, well, how far does that data go back? And then Mark said, well, actually, I think I've got, like, three months worth. So we kind of rolled this thing back three months. And there was one single device that had been hit, like, a month or so beforehand.

26:02Like, sometime in February, if my memory serves me right. And it was just really strange. So it was kind of registered to, like, a Chinese 163 address. And it sat again in Chengdu. Chengdu, China, again? That's where the person who submitted the bug was from. So they took this firewall. And again, this one was running a trial license, which was actually just a software-based firewall running in a virtual machine. And it's a virtual machine because Sophos isn't allowed to sell their firewalls to China due to export controls.

26:36So really, nobody in China should even have a Sophos firewall. Their suspicion was that the attackers were using this virtual firewall to practice their attacks against, develop them, and then unleash them against the world. Because Sophos has the ability to run in a virtual machine with trial licenses, they can just spin one up real quick, try attacks on it. If they mess up the firewall, they can just reboot it, take it down, and bring a fresh one up in minutes. We found this trial license, and they were kind of also associated to a 163 address

27:08and a moniker that we called GBigMow. Okay, interesting. They looked up who registered that trial license, and this gave them an IP address, a username, and an email address. And the username was GBigMow. So now you pivot on that name. What other Sophos products has GBigMow downloaded? We kind of pivoted on him. We found that he actually started to experiment with this database or SQL I injection,

27:39like our mother's so-go. And we kind of found then, looking at his IP address, again, we had phenomenal telemetry here, that he was looking at different knowledge base articles around our kind of previous CVEs, issues. He was looking through our forum system to look at maybe other potential issues or places that he could maybe pivot and work on. Then they took a look at his email address and wondered, has this email address been used anywhere else in the world? So they do some OSINT investigation to see if this email is known anywhere else.

28:13And we found that he was an actual firewall researcher. And he published like a number of different like vulnerabilities. We could see him on kind of Linux boards, you know, publishing various different router vulnerabilities. Up until about 2018, and then he went silent. You know, he'd been really, really busy up until like 2018. Now, we kind of found out that he was working for a company called Xizuan Silence Information Security Technology,

28:45mostly because doing some extra OSINT, we found that his username appeared in many like Chinese hacking groups and lots of CTFs, so like capture the flag type events, where he'd been registered towards this company as well. So we found kind of corroborating evidence from a couple of different places that this was the same guy in the same company, you know.

29:11Again, located in Shangdao in China. So we found a really clear picture of who this person was. Now, his external OPSEC was pretty good. You know, like you would not have been able to find him that easily. But because we could see the internal telemetry and get the license information, kind of connect the dots, we could actually pin these devices to him and his usage. But what we had to do at that point was find out more about these devices that were being used for research.

29:45We found that from the limited telemetry that we'd started to gather with the first hotfix, but what we realized is we actually needed more. Like, we really needed more detail, faster detail to like a greater depth to understand what these guys were doing. So we developed a kernel implant in-house.

30:09A kernel implant? That's a nice way to say it. I guess when the good guys make it, it's called an implant. But if the bad guys were to make it, it would just be called malware. But essentially, a kernel implant is a hidden piece of software that they developed to sneak onto their firewalls to covertly and sneakily spy on what the firewall is doing. Yeah. So there's a lot of interest within the company. Well, we know that there's these firewalls that have been registered to people who have non-corporate

30:41or non-enterprise-level email addresses, like free webmail addresses. The firewalls are checking in all from Chengdu. We know their serial numbers. So we know the exact count of the number of firewalls that are being used in these places. And we could see from some of the log telemetry that the threat actors are running commands that are testing how these exploits are going to work. But we don't have the exploit code itself.

31:12So the security team decides they're going to build something that they just call the implant, or sometimes they call it the kernel implant. And it's a small ELF binary that gets distributed only to the machines that they are specifically interested in taking a closer look at. So these machines that they believe are being operated by threat actors, where they're doing these commands that are way outside of the boundaries of normal firewall behavior.

31:44And these things are capable of doing more than just sending log entries. They're able to pick arbitrary fields from the file system on the firewall and send those files back. So that was how, in some cases, the team started throwing these kernel implants onto some of these firewalls that we could see were being used to do this experimentation.

32:14And they were retrieving all sorts of very malicious and pretty dangerous files that were being dropped on these machines by the people who were developing these exploits and were testing them out in advance of attacks. Wow, that is wild. This is going to take me a minute to fully grasp. Sophos developed an implant and sneakily put it on one of their customers' devices to essentially spy on them.

32:45Is that going too far? To call it malware is kind of a misnomer. I mean, I'm not going to defend the overall argument here, but I will just say that, like, there is nothing malicious about wanting to know what someone who is doing malicious things with your product is doing. You know, it's kind of a, it's an ethical gray area. I've got a caveat this with, we only ever deployed this to devices

33:17where we would be absolutely certain that they were a threat actor device. You know? And not just threat actor controlled, but threat actor owned. Like, this is where they're doing their research. Exactly. So, number one, like, we never deployed it to any properly licensed devices. The second part is, like, we only ever deployed it to Chinese devices. We just didn't sell firewalls in China. So, there was really, unless you were a company maybe bringing one from external, there was no real reason for you to actually have one legitimately in China.

33:48So, under the EULA, you know, we could take steps to, to protect the firewall, you know, and gather intelligence. And that was covered clearly under the EULA. Now, the other aspect... So, that's what, you got, you know, 40 people in the room, the lawyers must be in there too. Like, are we allowed to hack into these devices that we think are owned? That was a serious conversation we had, yeah. Yeah. I mean, it wasn't just a small one either. I mean, I don't think people have ever done this before, you know? Like, we sat there debating this thing for hours,

34:18you know, and really hours because, you know, there's some serious ethical challenges around this. You know, it's not... You know, what happens, like, if we find the guy, we, you know, we record him, we see him doing it and we send it through to law enforcement, you know, like a wee facility. You know, there's so many crazy things that we discussed there, you know? Yeah. It's a conversation that I never thought in my entire career that I would have, you know?

34:47Yeah. I mean, candidly, too, I never thought legitimately in my entire career that I'd ever deploy a kernel implant either, you know? But it was certainly interesting, you know? Well, I've never heard of a security vendor doing anything like this. Adding in stealthy secret implants to spy on their users? In my opinion, spyware is malware. And gosh, before hearing all this, I would have said, that is going too far. But now, now I'm not sure. My ethics are really being challenged here.

35:17And again, you know, I had amazing access to just quite incredible engineers. They built this kernel implant that allowed us to basically move softwares firewalls from like a normal update path to like a specific update wing. And we would then deploy this specialist kernel implant in a normal update. And you just wouldn't see it. But what it allowed us to do is like grab anything being needed from the device.

35:48So for example, things like, you know, files, if there were entry updates, it would kind of record anything that was kind of written to specific writable directories. And it would start to give us a really good idea of what they're doing, what they're writing, why they're doing it. But some of the really cool things that we actually got from it were quite unexpected. So for example, we started to pick up on the devices around the firewall. So we'd, you know, capture all the MAC addresses

36:19of devices connecting to this firewall. We'd also capture MAC addresses of things that also sat in the network alongside the firewall. And then we suddenly realized that actually this is huge. This isn't just like softwares firewalls. We've seen other vendors' devices on the same subnetwork alongside the softwares firewall. You know, they were looking at all sorts of devices. You can probably pull from the top of your head thinking about things that have been attacked in the past couple of years,

36:50the devices that were in the rack alongside that softwares firewall, you know? Oh, wow. So the firewalls that come to mind for me are like Cisco, Palo Alto, Juniper, Checkpoint, Fortinet. And he says he saw other vendor firewalls set up alongside their firewall in this threat actor's lab. Now, just being the, you know, person who's telling this story of what happened, we were observing, you know, in the world, not just Sophos firewalls, but every firewall vendor getting hit with zero days.

37:22There are customers being, you know, attacked in various ways. And there being no way to resolve this. And certainly no way to anticipate it. Now, whether or not other companies are doing the same thing, no one else has disclosed that. But I don't think it's outside the realm of possibility to think that maybe some of them were. Oh, man, this is now tugging at me in new ways. If every firewall vendor is getting hit with the same type of attack,

37:54and Sophos is the only one being transparent about what they're seeing and what they're doing to mitigate this, then yeah, I give them a lot of credit for that. Here's the test, I think, for whether your company is evil or not. First, it has to be transparent to its customers. Let them know exactly what kind of configuration changes, updates, or spying, or data collection you're doing on your customers' devices and in what circumstances and what's that you're being used for. And second, be proud of whatever it is

38:24you're doing around that. If you're a company which is making changes to the customer's products, but then not telling them and secretly adding spyware, but making it so top secret that not many people on your team even know it exists, then I think you might be evil. If you're afraid to let the public know exactly how you operate, because you think it's going to look bad on you, or maybe because you think it's not even right, then either stop doing it or go public with it. And Sophos came to the conclusion that while this is not

38:54an ideal situation, this threat is novel and sophisticated in ways nobody's ever seen before. And not only that, whoever was doing this, they're being unethical themselves. So Sophos had to deploy a novel and sophisticated approach to defending their device. And while it's not pretty, at least they came out and told us about it through Andrew's blog posts. And they're basically saying, hey, we're in the middle of a nasty street fight here, and the gloves are off until we can neutralize this threat.

39:26And again, I give them a lot of credit for that. Nice job. So at the same time, they were developing this implant to eavesdrop on the hackers. They were also in the process of studying those domains which were found in the exploited firewalls. The hackers pointed all the firewalls to two domains to get updates from which were not owned by Sophos. Yeah, well, there was SophosFirewallUpdate.com and SophosProductUpdate.com which were registered at different registrars and hosted in different IP spaces.

39:57But because they were, they both had Sophos in the name and they were part of this attack, Sophos went to ICANN and did the domain name seizure process on those domains so that they could pull those down and start to, they wanted to sync all the domains and see what was connecting into them. How do you seize the domain? Well, with lawyers and money and, you know, it's a really serious thing, you know, like attending court

40:27in Delaware, I think it was, you know, remotely because at the time, don't forget that this is the thick of COVID. Jeez, that's another thing that's wild to me. The fact that you can take over someone else's domain if you can prove that you're the one who's the rightful owner of it or should be owning it, but they gave enough reasons to the courts who then demanded that the domain registrar give Sophos control of the hackers' malicious domains.

40:53The server used by the threat actor actually sat in the Netherlands and it was one of these bulletproof, like, hosting providers. So we were super lucky that, you know, through the NCSC in the Netherlands, they were kind of an intermediary with the kind of Dutch National High Tech Crime Unit and once we kind of realized how this was panning out, the Dutch National High Tech Crime Unit just jumped on this and they managed

41:24to get hold of this C2 server, so the actual physical Linux box. I guess it wasn't bulletproof then, huh? Well, yeah, this is the thing, you know. So they managed to grab hold of it and, I mean, we were super keen to... How do you even... So how does that happen? You convinced the Dutch authorities. So you're just a company in the UK. You're just like, hey, we make this product. You can't just call up the Dutch police and say, go get that server, we need it and then they're like, we're on it. Well, yeah, I mean, you'd think but then, you know,

41:55luckily or unluckily for us, there were a couple of Dutch customers affected, you know, by this attack. So that allowed us to be able to register a crime and then get assistance. And we did this globally, you know. We really used all of the resources available to us.

42:15So, you know, this obviously took time. You know, I think right now this is like three or four days after the attack. But the NCSE in the Netherlands were incredible and the Dutch guys there were just super helpful. I mean, we wanted a copy of that threat actor device. Like, I wanted to see that Linux box and understand what they've done. I mean, obviously, it was evidence now. It wasn't owned by us. So we couldn't get a snapshot of it, for example.

42:46But they allowed us to basically, you know, work with them and analyze the box live on a screen share so we could actually understand the scale of what had happened, you know. And we'd seen the threat actor scripts for scanning the devices, the outputs that they'd taken from the firewall, you know, how they'd set this thing up, you know, kind of Chinese characters and notes and things throughout the device. What was actually surprising was that everything was kind of set up

43:18manually on the C2 server. I kind of expected them to deliver the C2 server with some sort of kind of DevOps pizzazz. But it was just basic. You know, it was like a Linux box and someone had copied subscripts to it, you know. But they were amazing. I mean, the NCSC in the Netherlands just gave us so much help and really helped us focus what we, you know, where we needed to look and the kind of scope and scale of all of this. At the same time, they got control of the domains used by the hackers

43:48and sent all the traffic they were getting to a sinkhole and logged it all. It's just fascinating to think that like, I don't know, a Netgear, a Linksys, you know, some other commercial product was checking in to SophosFirewallUpdate.com. It kind of, it's, it's almost screams of like, well, you know, we could be bothered to register this domain for Sophos. We're not going to bother to register it for these other companies. Like, we already got the domain. and we're just going to keep using it for these other things. I couldn't find

44:20a single article by Linksys mentioning any of this. Nothing at all. Netgear put out an advisory saying a Chinese threat actor is attacking their products. However, they say they are not aware of any Netgear devices being exploited out in the wild. Which, if they don't have any telemetry from their customers' products, then yeah, of course, they're not going to know if any devices are being exploited. And that's what's challenging me here. Should the firewall vendor be collecting logs off its customers' devices

44:52in order to better understand what devices are actively being exploited? Or should that be the responsibility of the customer? In many organizations, they have their own security logs and even a team to monitor those logs to look for threats. But things like Netgear and Linksys are typically home devices and it's very rare for people in their own homes to be monitoring their logs looking for threats.

45:16I looked it up. Netgear actually does quite a lot of analytic collection from their customers' devices. They collect IP addresses, geolocation, how often you use the firewall, what you use the hardware for, what channels your Wi-Fi is set to, and what devices are connected to it. It's surprising with all that analytics collected that they didn't spot a single device being exploited by these threat actors. And this is what frustrates me. When my home router is sending all kinds of logs to another company, like what devices are connected to my router?

45:47Really? I hate that. I want the devices in my home to be private and not sending tons of data to somewhere without me even knowing. Because if Netgear has that data, then it's likely a lot of other people have it too. But then they also registered for the kill switch, they registered Ragnarok from Asgard, right? And Ragnarok, of course, is the Norse mythology end of world myth. And it was fascinating that that was how they, you know, used that nomenclature and that language behind it.

46:19Because by this point, we already had some folks who were using Marvel characters, superhero names in their user accounts that they were, you know, that they were using for downloading these firewalls. So we had a guy who used the handle of T. Stark, who was involved in some of the exploit development and had registered a bunch of these virtual firewalls. And now we're seeing, you know, this is the time frame when the TV series Loki came out