Threat Hunting with Active Defense: Q&A with Matthew Hosburgh
Matthew Hosburgh is a Marine Corps veteran and a current analyst at Radian. He has over 13 years of experience working with various systems, networks and security disciplines. Matt has worked as a Principle Security Analyst for United States Citizenship and Immigration Services (USCIS). Matt has also worked on securing SCADA and other business systems for MarkWest Energy Partners. He is a candidate in the Master of Science Degree Program of the SANS Technology Institute, and holds numerous certifications.
- “Active Defense” can be broken into three parts: annoyance, attribution, and attack. Each of these three can be used to make your business a less attractive target for attackers
- Responses can be grouped into three categories: automated, semi-automated, and human-driven. You’ll gain some of your most valuable insights from finding mistakes or unique actions undertaken by attackers.
What are the best ways for hunters to take the fight to an adversary? How should analysts sort out which tasks can be automated verses those that require human attention? In this interview, we sat down with Matthew Hosburgh from Radian to discuss these questions and more.
This interview was originally posted in conjunction with the Threat Hunter Spotlight series which features conversations with top-level threat hunters to discuss a range of topics, from spotting adversary tactics, techniques, and procedures to leading hunt teams. Matthew’s original “Threat Hunter Profile” can be found on the Sqrrl blog. The original interview is available here.
Question: How did you start Hunting?
Matthew Hosburgh (M): So, I am a cyber threat hunter for a company called Radian based out of Philadelphia and been there for several months now at this point. I’ve been doing incident response, threat hunting, security analysis for the past, roughly seven years or so. I like to think that threat hunting, nowadays, is something that’s kinda become a buzz word. However, it’s a technique that’s kinda been around for a while. It’s something I’ve been doing for several years and I’m kind of excited to see the industry kind of embrace it a little bit more whole heartedly
Q: When is a good time for analysts to start hunting?
M: As a threat hunter, I would tell you that you could start hunting basically right now. You could hunt on anything from external facing issues that your organization has. You can hunt on internal anomalies within your organization. You’re gonna get better results over time and what I mean by that is think of it if you’re a home inspector. When you’re buying a new home, most people have to get their home inspected. You hire that inspector, they come out into your home, they run through in a couple hours and provide a nice little report for you. Threat hunting, maybe in one aspect, is kinda like that, where you don’t have a lot of information.
But, let’s say you had more time. You are able to determine what is normal or abnormal within your network or system, it’s almost like being a home inspector for your house that you’ve lived in for 20 years. And I think, obviously if you look at the two, the one that doesn’t have the experience with maybe the systems, you’re not gonna get as much. You’re probably still gonna always find something. But, the more mature, even the more kind of time you have with that network or system, obviously the more you’ll be able to identify some of those issues that might be hard to uncover otherwise. So, I always tell everybody it depends on what level you’re trying to get out of it.
Q: What categories do you personally like to hunt on?
M: Well, generally, I break those up into three major types and these are kind of the overarching categories. But, the first one is your environmental variables. So, this is looking at things that are either normal or abnormal within your environment. For example, when looking at an SPC host process, you look to see if that’s being spawned out of the right directory, and also maybe with the right flags there. And if it’s not, that’s something you could potentially hunt on. And then, again, back to the point of if you have maybe a massive EDR solution for example, or way to pull all your devices, that task might scale better that way. If not, it might be a more manual process.The second one, though, I like to look at are the attack patterns. And some of you may be familiar with kinda the minor attack technique matrix. That’s a great place to start to build a hypothesis for a hunt because those are mapped to real techniques seen in the wild, used by advance adversaries.
And then, also, another great resource if you haven’t checked it out is Threathunting.net and that actually will give you some ideas of where to look. For example, they might show you, if you’re looking for lateral movement, some event ID’s you can check out and kinda drill into. Again, back to the point of maybe you need to set that logging up first, or auditing on the actually system itself. So, and then, finally, and kinda what we’ll be talking a little bit more about is threat intelligence. Now, when I say that, I know that’s kinda loaded and a lot of people have a lot of opinions about it, but for our sake of the conversation, I’m talking about kind internally generated threat intel. So, things you can gain yourself, whether that’s from a honeypot, active defense techniques, which I’ll mention and then, also the other side of the corner is your external thread intel.
Q: How can techniques like that be used for active defense?
M: So, active defense, really, is truly a phrase coined by John Paul Estadorian. They wrote a book several years back and they’ve kind of outlined what it is. And essentially, there’s three A’s, I think there’s a fourth one at this point, but the main ones are annoyance, attribution, and attack. And so, if you think of it like a continuum of force or maybe the level of interaction you have with an adversary or someone trying to get into your network, it overlays pretty nicely. So, let me just break it down: annoyance is obviously something that’s going to annoy or frustrate an attacker. So, the concept there is to really make yourself a hard target to move them on to another, maybe a lower hanging fruit target. Attribution would really, truly be trying to identify who is attacking you.
And that, again, has kind of a spectrum. So, attributing back to maybe a country level, or an organizational level, or even down even to the individual level is kinda where that resides. And then, finally, and one of the maybe the areas that’s very controversial at this point is attacking back. But, again, I’ll be talking about, really, just kind of an attribution perspective. So, those are kinda the three that are within the active defense umbrella so to speak.
Q: What would some examples of that be?
M: Yeah, so that second part is how I’ve leveraged that in threat hunting, or at least, how do I see the value of that? And so, I’ll give you an example. Most of us, whether you’re a security professional or not, are pretty familiar with phishing and other social engineering attacks. They’re very common, one of the most common attack sectors these days. Check out any report and it will line up pretty nicely with that. From an active defense perspective,we’re gonna kinda turn the tables and leverage the human element or even some of the aspects of social engineering to interact with our attackers.
One of the tools they’ve kinda packaged up with in there is called Web Bug Service. And all that is is a simple server that can exist on an internal network or somewhere out in the cloud based environment. And basically, what it does is you would take a document, we’d place a simple web bug in it. And so, what a web bug is, it’s a one by one image that has a link into it. So, when it’s referenced in the case of a document, when the document is opened, it’ll send a beacon out to your web bug server. And so, you might be saying, “Well, that kinda sounds like a bad guy technique. How do we use that for threat hunting?” Well, in the concept of attributing … Or, attribution and focusing on how to find, let’s say, an insider threat, taking these documents that you’ve bugged and just placing them throughout your network or organization in areas that may not be super obvious.
But, if somebody were poking around where they shouldn’t be, this could be an early indication that somebody’s looking for maybe sensitive documents or things, just maybe out of curiosity. Where I see the big value of this, though, is so, you may have an early indication there. But, if you see these documents somehow starting to show up with an external based IP, so back to if you had a web bug server out in the cloud. Instead of seeing it just from your organization’s IP infrastructure, you start to see it beacon from random IP’s. It’s very suggestive of documents that are being leaked within your environment. And so, really, the concept comes all the way back to early indication and now, if I start to see things call back to my web bug server, I have a very strong idea that there’s a … Or very strong hypothesis that there’s an active insider threat.
And so, based on that, let’s say those documents have a unique ID that you can tie to them. At least, it’ll show up in your web bug server. You can go back and say, “Okay, maybe the documents I staged for marketing, I saw two of those and maybe I saw one over from our networking team,” for example. Based on that traffic you saw, you could go back and use another tool called Mole Hunt and build out specific lists of potential insiders. So, that ID is tied to a user so that if that document is leaked or shows up on your web bug server, now you have a very, very high indication that that particular user is the insider. Now, I know the question that is probably on your mind is, “Well, how do you know that’s not some malicious code on there running and exfiltrating?”
Well, I don’t necessarily based off of that. But, what I can determine from it is that I can focus my hunt efforts on those particular either users or set of users. And from an organizational or security operations perspective, you can step back and say, “Okay, let’s put some additional monitoring. Let’s see if we can prove that hypothesis and,” or maybe that’s enough for your organization. Maybe you say, “Okay, cool. I know I have these insiders, I’m going to escalate that up to somebody else that can deal with it.” And really, the hope is you’ve caught the insider before they’ve leaked the real stuff. So, you might make these kind of enticing. But, you make it so enticing that it could be included in that leak so to speak.
But, the other thing that I always hear, too, is, “Are we entrapping our users?” And so, I’m not a legal expert, so you definitely wanna vet this through a legal department if that’s your plan, but just a really cool way, I think, from a threat hunter perspective to kind of go after those adversaries on your network ’cause threat hunting truly is kind of the proactive identification of either intrusions or adversaries on your network. At least that’s the way I kinda like to look at it, so.
Q: What is the human element in threat hunting?
M: So, you can think of it as a pyramid. There’s three levels– It’s widely practiced, that you have a bottom part of the pyramid that is your automated process. A step from that is your semi automated, and at the very top is our manual process or analysis. And so, a threat hunter’s regime, so to speak, can benefit from all layers. For example, I can automate my indicators with compromise searches. That’s pretty easy. I can ingest threat intel that’s already been produced, throw it in there, it’s automated, so I get an alert, then I can drill down.
Semi automated might be something similar where I could detect a little bit of a pattern, but I might have to drill down manually just to confirm or deny. And then, at the very top, which I think is probably where you wanna focus on. That’s where you’ll get the most value. I always approach it from whatever I do manually if I can find the repeatable tasks and either take them back down to my automated process, so I can ingest them into an existing instant response maybe workflow. Or, if I can take parts of that to kinda get it off my plate, and if you don’t have a mature security operations or way to handle alerts, okay, maybe that’s somewhere you need to focus first.
But, again, I always try to think of it in those terms because I really wanna focus on those human, non repeatable things. So, if I can find a clumsy attacker maybe, they’re gonna make mistakes. But, if I find an attacker that leveraged the script, it’s gonna be really fast, it’s gonna be very hard to detect from a human perspective. So, if I can kind of send those faster things down the stack, so to speak, to a faster computer or system to detect it, then that’s always my on the forefront of my mind.
Q: What advice do you have for people who are just starting out with hunting?
M: I think the first question you wanna ask yourself a question is “would you like to be the pilot or would you rather be the aircraft mechanic or engineer?” Is it more of a passion of yours to build systems, to get them working, and then allow somebody else to look at them like an analyst or threat hunter. Or, do you like to be in that pilot’s chair flying the plane and you don’t really care how it works necessarily, but you care more about what the content is.
If you like that, I would say start jumping into packet analysis or just other ways, get your hands dirty, like I said, with Security Onion. And start looking at attack patterns. And then, test them out, apply them into a lab environment, and see if you can hunt on them. And if you like doing that, you’ll love threat hunting. If you hate that, you might struggle. And so, maybe that’s the biggest question to ask yourself first off. But, I also did some security engineering in the past and I guess that’s why I like threat hunting at this point because I can still go build things in a lab, test it out, but I don’t have that large burden of keeping all the firewalls running or something like that. Which, isn’t a burden, it’s also fun. But, it’s more fun to find bad guys on a network.
For further information:
- View Matthew Hosbourgh’s Threat Hunter Profile.
- Read his post on examining network patterns hunting Powershells.
- Check out his post on examining the host hunting misbehaving PowerShells.