Threat Hunter Profile – Bilal Malik
DNS, HTTP, Proxy, connection logs and Windows endpoint logs
Preferred Hunting Techniques
Visualization, Outlier analysis
Unix command line, custom scripts, Tanium/Carbon Black, Bro, ELK/Splunk, RITA
Who are you?
My name is Bilal Malik, and I am a Senior Consultant at Stroz Friedberg’s Cyber Resilience unit.
What is your experience with Threat Hunting and for how long?
I am a lead threat hunter at the firm and have been hunting for about 3 years now. I have helped various clients plan, implement, and scale their Threat Hunting programs from scratch. This involves assessing and making technology recommendations as well as developing mature hunting processes, and training security analyst teams to become self-sufficient in hunting techniques.
Why do you hunt and/or how did you get started?
At Stroz Friedberg, I primarily perform IR investigations on the front-lines to help organizations contain breaches and advise on ongoing remediation efforts. As part of that remediation effort, we identify most clients lack Threat Hunting, or even basic SOC monitoring capabilities. That’s typically the norm across the industry. Everyone has been focused on being overtly preventive by investing a lot in preventive appliances and hoping it works, but in-reality prevention eventually fails and most clients simply lack visibility and breach detection capabilities.
It’s becoming quite obvious that data breaches are inevitable, and clients are, slowly but surely, trying to develop Threat Detection capabilities, whether with the help of MSSPs or internally, to detect attackers early and reduce dwell time. Given my understanding of adversarial tactics, Threat Hunting came naturally with the goal to kill the kill-chain in its early stages. Perhaps one of the greatest joys of doing Threat Hunting is the satisfaction you get when you enable an organization to be self-sufficient in their hunting programs and they are able to identify threats early and independently, before being notified by an external party.
I would say, it just came naturally for me and the satisfaction of helping clients motivated me to continue to innovate further in this space.
How would you define Threat Hunting?
In very simple words, I would define Threat Hunting as a human effort to detect post-exploitation activity that other tools may have missed.
Let me ask, what is common between Firewalls, AV, IPS, DLP, NAC, Proxy? They are commonly used traditional cyber defense tools which are all overtly preventive in nature. Modern attacks are novel, sophisticated and pretty dynamic that it’s simply not possible to be able to prevent them completely like we have traditionally approached security. We need a paradigm shift in the way we approach cyber defense and consider developing tailored tools, techniques and processes to develop detection capabilities against dynamic modern attack techniques. Threat Hunters look for threats that tools miss.
Threat Hunters spend time understanding network data flows because one-size doesn’t fit all, and not all networks are created equally. Threat Hunting involves the human element to manually observe data flows and identify abnormalities or signs of intrusion and post-exploitation activity like lateral movement, data staging and exfiltration.
What projects and organizations are you involved with right now?
I am an active member on the SANS advisory board and have performed well in several capture-the-flag competitions over the years from SANS and Symantec. I work with several local universities in Chicago to help mentor and advise students interested in starting a career in Cybersecurity, including Northeastern Illinois University, DePaul University, University of Chicago and Northwestern University.
Which of the hunts you’ve carried out was the most interesting or challenging?
It’s hard to say which one was the most interesting because after hunting for a while, everything begins to look normal when you see it again and again in different environments. What I really find interesting is collaborating with other hunters to share ideas on how to hunt for various parts of the Mitre ATT&CK Matrix. I like innovating new methods to keep up with the various phases of an adversary’s lifecycle.
I will share an example of something I found from a recent hunt. While performing simple egress analysis, I identified suspicious traffic. Upon a closer look, I determined it was beaconing activity with encoded parameters in URLs. Since I know the environment pretty well, that traffic was unexpected from that part of the network. Luckily, the client had a robust architecture with full-network and endpoint visibility which enabled me to pivot to the endpoint and identify a malicious DLL injected in memory performing keylogging and beaconing out to a CnC.
What hunting techniques, tools, and datasets do you use most frequently?
Attackers will definitely try to persist within a network and they are not going to get much done without a C2 channel, so I really like to start with a focus on egress analysis reviewing what’s leaving the network and from where. Firewalls or Bro logs can be really helpful in identifying persistent outbound connections, beaconing activity and even inside users trying to bypass security controls through unauthorized VPNs, tunnels and proxies. There’s just a lot of value in egress analysis and you are always bound to find something. I have some custom scripts to look for beaconing activity from Firewall or Bro Conn logs.
Reviewing DNS logs for high entropy domains and HTTP logs for naked IP addresses are some easy wins to look for attacker activity. Reviewing traffic to domains other than Alexa 500 can help shine some light into the dark areas of your network and reveal domain names used by attackers or malware.
A lot of attackers are laying off the land and utilizing tools built within Windows like command-line utilities and Powershell so I look for anomalies in endpoint logs to look for strange or encoded Powershell behavior which helps identify users that may have been spear-phished with macro attachments. Reviewing usage of command line administrative usage on endpoints can also be a sign of an attacker performing reconnaissance or provide insight into their persistence mechanism.
What value do you actively see come out of your hunting activities?
Clients are able to identify when they are attacked with malspam attachments before it gets worse. Threat Hunting has enabled organizations to identify persistence mechanisms when they are created. Threat Hunting endpoint logs for process arguments has improved visibility into post-exploitation activity. We are also able to identify C2 communications early and often to initiate IR handlers before things get worse.
What types of friendly intelligence are most useful for a hunter to have in an investigation?
Every hunter should start with an inventory of their network and identify what critical assets they are trying to protect. Knowing how many systems are in scope or what you are trying to protect, helps put things in perspective. Hunters should have an idea of how many different sites are in scope and what software is authorized or unauthorized. Having a general idea of which IP subnets belong to which business units within an organization will help you map your defenses accordingly. Hunters should identify different Egress points and classify different network segments based on sensitivity of classified data.
Aside from knowing their technical environment in-and-out, Hunters shouldn’t lose focus of the bigger picture. Take a moment to identify your company’s “crown-jewels” that the attacker may come after and exactly where they are stored. It could be your organization is targeted for intellectual property (IP) theft so you need to identify which systems store IP. Not all attackers are after IP, some attackers just want to cause disruption or denial-of-service, so it’s also highly recommended to think about the attacker’s motivation to anticipate and be prepared accordingly rather than getting worked up with technical details.
What general advice do you have for new Threat Hunters?
Start reading reports on industry trends to get an idea of what types of attacks happen in which industry, and begin by asking yourself basic questions if you are prepared to be able to detect and prevent those kinds of attacks. Take a look at your toolset and identify areas that may need more visibility. Learn the various phases of an adversary’s lifecycle using Mitre ATT&CK methodology to identify gaps in your monitoring strategy and come up with ways to look for those gaps. Last but not least, don’t be afraid to collaborate and ask for help if you lack in one area or another. There’s plenty of resources and helpful hunters that can point you in the right direction.
What hunting procedure would you recommend for a new to Threat Hunter?
I would recommend new hunters to start off by analyzing user-agents in their environments as it’s pretty simple to do with Bro logs. Looking for unexpected user-agents can help grab some low hanging fruit and possibly discover commodity botnets pretty easily. As you begin to feel more comfortable, start implementing additional procedures by following ThreatHunting Project by David Bianco.
What parts of a hunt could you see as being most successfully automated or assisted by a machine?
With the power of machine learning and AI, there’s great potential for technologies to assist in identifying unusual use of user accounts against unexpected systems. This sort of technology could help assist in identifying lateral movement early in the kill chain. Advanced algorithms can also assist in identifying beaconing or covert DNS tunnels. I also want to emphasize the “perfect tool fallacy” that while technologies can assist, they are not always perfect so its recommended to understand exactly what the machine is doing and identify any remaining blind spots that may go undetected.
What would you like to see Threat Hunting develop into across the industry in the future?
The Threat Hunting industry is new and in its early stages. There’s lots of talented hunters coming up with novel techniques every day that they find useful. It would be nice to connect and collaborate with others to share what worked well and what didn’t. The ThreatHunting Project is one great initiative in that regard that I am really excited about!