Threat Hunter Profile – Chris Sanders
Applied Network Defense
Flow, Bro, Windows endpoint logs
Preferred Hunting Techniques
Aggregations, pivots, relationship graph visualizations
SiLK, FlowBAT, Python, Wireshark, FireEye TAP, Splunk
Who are you?
My name is Chris Sanders, and I’m the founder of Applied Network Defense, an information security practitioner-focused company dedicated to developing and providing high quality, tailored training and education.I blog at http://www.chrissanders.org.
Why do you hunt and what is your experience hunting?
I’ve been hunting in one form or another for about 10 years. I first started hunting when I was a consultant back in my home state of Kentucky, but I didn’t really get serious about it until I started working for the US Department of Defense, helping to build a SOC enclave for the Army Research Lab. That was the first time I had access to a massive data set to really go crazy. The fire hose had been turned on!
I mostly hunt now for two purposes. At FireEye, where I worked in the past, we often were the first to know about certain types of malware or attack techniques because we have many incident responders out on the front lines. Those folks fed intel back to us, and we used that to drive our hunting efforts. In addition, I spend a lot of time hunting for research purposes. A real goal of mine is making hunting more accessible for the masses. We have a real lack of skilled practitioners, so I’m conducting research related to the cognitive underpinnings of the hunting process (and other investigative processes). The more we can understand about how hunting works internally and what skills are really required, the better equipped we’ll be to train more hunters. It’s all about taking the tacit knowledge of security investigations and turning it into trainable, repeatable, explicit knowledge.
How would you define Threat Hunting?
Cognitively speaking, hunting isn’t really that different from a typical alert-driven investigation. In most cases, the only difference is that in threat hunting you often don’t have an alert to start your investigation. Instead, you start an investigation based on an observation you drive from your own experience. So, in short, I’d say that hunting is an investigation that begins with a human-driven observation.
What projects and organizations are you involved with right now?
Alongside my work at Applied Network Defense, I’m also an author and enjoy writing and teaching people about threat hunting and security. I’m currently editing the third edition of my book Practical Packet Analysis which is set to release later this year. I’m also actively involved in writing my dissertation on how simulation-based training can be used to teach security investigation concepts, and I teach a university course on information security.
In addition, I founded the Rural Technology Fund in 2008. The RTF is, a nonprofit that advocates technology education in lower income and rural areas. We provide scholarships to students from these areas, and also outfit classrooms with technology resources like 3D printers, electronics kits, robotics kits, and Raspberry Pi’s to teach students how to code. This year alone we’ve donated $30K+ to classrooms across the country and helped to build makerspaces in these schools. If we want to keep fighting the good fight we need to be thinking about how to educate the next generation of threat hunters.
Which of the hunts you’ve carried out was the most interesting or challenging?
I’ve seen enough of real bad guys now that not much really gets me excited unless it has some kind of real world kinetic manifestation. That said, I have been part of investigating a few incidents that ended up leading back to internal users. This is rare, but I’m ultimately a lot more interested in the human aspect of security so it was neat to track actual humans that I could put a name and a face to.
What hunting techniques, tools, and datasets do you use most frequently?
The investigation is all about asking questions to uncover relationships, so I look at tools and data simply as ways to find answers to my questions via evidence gathering. Ultimately, I’ll take any data source which allows me the evidence I need.
It’s a general rule of thumb that hunting performance is better when you start with a smaller data set and expand it until you’ve truly gathered what you need (rather than the opposite of starting big and filtering down). With that said, I really like data sources that let me hone in on a particular time range quickly before pivoting to a higher context (but slower to retrieve/analyst) data source. For example, I generally start network investigations by looking at flow data (via SiLK + FlowBAT). This let’s me scope the investigation and ask more specific questions before I pivot to something like PCAP or a host log. Another great source of data with low retrieval/analysis overhead are Bro logs. When these are available I’ll often start there before pivoting to something richer or harder to retrieve and analyze. Threat hunting is an economic challenge because it is very time consuming and can be low reward over the long haul. The more those processes can be optimized and sped up, the better.
What types of friendly intelligence are most useful for a hunter to have in an investigation?
I love this question because friendly intelligence is often the most undervalued data source I see. For whatever reason, people will spend hours researching unknown external threats, but they never seem to want to investigate their own systems and baseline them.
For friendly intelligence, it’s all about establishing a norm. I like to know who a particular device’s friends and family are. Friends are who it talks to outside the network often and family are who it talks to inside the network often. If I know those things, it’s easier to spot suspicious communication.
Beyond that, I really think it’s important to focus on the human user of a system. For any given system, if I know a lot about the human that is using it I am much more likely to be able to tell you if a particular behavior is expected. What dept does that user work in? What hours do they normally work? Do they normally work from home and come in over the VPN? Do they travel for work? Answers to any of these questions are handy when you find something weird relating to that users system.
What general advice do you have for new Threat Hunters?
Success in hunting is really all about curiosity. If you see something that looks weird, find the motivation to run it to ground. You might not be able to fully explain it, but as you do find things like this you’ll eventually build a mental catalog of heuristics. That is, quick rules you rely on to indicate whether something is suspicious or not. The truth is that “gut feeling” is really not about your gut at all, it’s about your experience and the number of scenarios you’ve encountered. The more you let your curiosity drive you into new scenarios, the more heuristics you’ll develop and the more you’ll start developing your own “spider senses” about whether something is suspicious or not.
More tactically, at any given point while hunting you should be able to articulate what question you’re trying to answer. Otherwise, you’re hunting aimlessly and that isn’t fruitful. Try to start articulating these questions aloud to check yourself — you’ll be better for it.
What hunting procedure would you recommend for a new threat hunter?
The easiest way to get started is to take an easy data source you understand and start examining specific fields within it. For example, most analysts understand HTTP well and have that data source available via proxy logs. Pick any field! User agent is a good one. What can be in this field? Look on your network and count every user agent you observe and sort them. What are the most frequently seen UAs? What are the least frequently seen ones? Can you explain them all? This is a simple technique but it can be used across many data types and is very effective for finding evil. I look at the user agent field very frequently and have found a lot of evil that way (both malware, and human attackers). The key is to just start asking questions. Questions will beget answers which will beget more questions.
What parts of a hunt could you see as being most successfully automated or assisted by a machine?
“Mise en place” is an old chef term that means “everything in place” and is a principle chefs use to organize their work area. It’s critically important that an analyst masters their environment, and particularly, how they access data. I mentioned previously that investigations are all about asking questions to uncover relationships. Any time an analyst spends asking questions or analyzing data/evidence to derive conclusions is time well spent. Any time an analyst spends retrieving data is time wasted and can be automated. The more we can automate quick data collection to support answering investigative questions, the better.
What would you like to see Threat Hunting develop into across the industry in the future?
I think our industry has glorified threat hunting a bit and made it out to be a very elite skill only experienced practitioners can be successful at, and that makes it inaccessible to many. The supply of experienced hunters is low right now and demand is high. This has generated an economics problem where security is still WAY too expensive for small and medium sized businesses, and as long as that issues exists we’re going to continue to get beat by the bad guys. We’ve got to take a hard look at how we educate practitioners and make that training readily available (and affordable) so we can close this gap.