Week 8 – Advanced Pen Testing Paper
Hunting takes cybersecurity to the next level by making it an active process in which security analysts sniff out traces of cyber attackers and go in pursuit, relentlessly tracking and hunting down their prey (Ashford, 2015). In anticipating opposition to cyberattacks, organizations can build stronger defenses, because they can find and fix vulnerabilities in their networks and systems before they are attacked maliciously. Proactive defense is key to mitigating operational risk, because cleaning up the aftermath of an attack is much more costly than proactive defense strategies.
Hunters typically look at all processes, tools, commands, and network file shares that are running in an environment to find potential vulnerabilities that typical security systems, like firewalls, antiviruses, etc., would miss because they are not malicious in and of themselves, but a trained eye can recognize if something is inappropriate, unlikely, or unusual, which can signal that something is wrong. According to an interview by Computer Weekly, Ben Johnson of Bit9 + CarbonBlack says that this innovation in cybersecurity arose because large, well-resourced companies are getting hacked on a daily basis (Ashford, 2015). Because attackers are always innovating and evolving their capabilities, there must also be innovation and evolution of defense capabilities. Hunting typically involves the most enthusiastic, passionate, and security driven security analysts, because it is these individuals that enjoy proactively investigating and not waiting for alerts or emergency calls to come in. They know how to think like an attacker, act like an attacker, attack like an attacker, how to communicate with the attackers, and the good ones can even infiltrate cyber criminals’ minds and organizations to learn their techniques and find out what their plans and deeds are. For example, according to another article by Ashford, many hunters that work for security companies, such as RSA FraudAction, do this, and are long-standing members of hacker forums, talking directly to hackers (2016). This kind of proactive security is a bit extreme, and as such, these actions are carried out by only the most dedicated hunters. At the most basic level, hunters are looking for abnormal, unusual or suspicious behavior, especially in relation to high-value data assets, wherever there is risk and attackers may be active (Ashford, 2015), which could be anywhere on a network, at any time, with or without real login information or administrator privileges.
One of the reasons hunters must exist and are in high demand is because attackers can mask their attacks to look like normal network and/or system usage, which doesn’t get flagged by automated security systems. For example, when an attacker steals valid user credentials and uses them to log on to a network or network device, it is difficult to detect them because there is no malware or malicious code; it simply looks like a user has logged in to their account. A hunter would look for multiple logins at the same time. A hunter could look for the terminal or command line command to pull password hashes into a file, like the bkhive command, which dumps the syskey bootkey from a Windows system hive, and the samdump2 command, which dumps Windows (up to Vista) passwords and hashes. This command is not a command that a typical user would know, and so a hunter could collect all processes and commands running on all endpoints of a network, making it possible to identify compromised computers by tracking commands, like the aforementioned Windows command, that most people don’t know about.
Advanced persistent threats present significant challenges to the security community and changes how organization need to view, implement and manage security operations, according to Rackspace (2017). Advanced persistent threats occur when attackers capable of breaching data infrastructure through continuous targeting, and then remaining within that infrastructure, undetected, to locate and access valuable information, and as Daniel Clayton, a former British intelligence officer who now serves as a director of security operations at Rackspace, describes, advanced persistent threats are typically “groups of individuals that have the resources and manpower to persistently target a company or organization 24 hours a day for as long as it takes to get the job done” (Rackspace, 2017). While prevention measures, like web application firewalls, intrusion detection and preventions systems, and anti-virus software, can be effective against some attacks, like DDoS, viruses, Trojans, and other attacks that remain consistent across all platforms, the reality of advanced persistent threats has made many of these measures obsolete in the modern world of cyber security. Effective security now requires firms to assume penetration and continually and actively scan their environments for malicious activity.
Modern security providers deploy sophisticated technology and highly skilled analysts to actively patrol environments and locate anomalies. Cyber hunting is a focused and iterative approach to searching out, identifying, and understanding adversaries internal to the defender’s networks, according to Lee (2016). The formal process of threat hunting should not be confused with an attempt to prevent adversaries from breaching the environment or for defenders to eliminate vulnerabilities in the network (Lee, 2016).
There are three factors to consider when judging an organization’s hunting ability: the quality of the data they collect for hunting, the tools they provide to access and analyze the data, and the skills of the analysts who use the data and the tools to find security incidents. Bianco describes a hunting maturity model based on primarily the skills of the analysts, because they are the ones who turn data into detections (2015). The quality of the data that an organization routinely collects from its IT environment is also a strong factor in determining the HMM level. The more data (and the more different types of data) you provide to an expert hunter, the more results they will find. The toolset for collecting and analyzing the data is a factor as well, but a less important one. Given a high amount of analyst skill and a large amount of good quality data, it’s possible to compensate for toolset deficiencies, at least to a degree. The hunting maturity model ranges from HMM0, the initial stage of maturity, in which, an organization relies primarily on automated alerting tools, such as IDS, SIEM, or antivirus, may incorporate feeds of signature updates or threat intelligence indicators, and routinely collects little or no data, to HMM4, the leading stage of maturity, in which an organization automates the majority of successful data analysis procedures and routinely collects high levels of data. In HMM4, organizations will turn any successful hunting process into operational, automated detection, which frees analysts from the burden of running the same processes over and over, and also allows them instead to concentrate on improving existing processes or creating new ones. This makes HMM4 organizations extremely effective at resisting adversary actions, by allowing them to focus their efforts on creating a stream of new hunting processes, resulting in constant improvement to the detection program as a whole (Bianco, 2015). The difference between the automation that HMM0 and HMM4 organizations carry out is that HMM4 organizations always have automation in the front of their minds as they create new hunting techniques, whereas HMM0 organizations rely entirely on their automated detection, whether it’s provided by a vendor or created in-house. They may spend time improving their detection by creating new signatures or looking for new threat intel feeds to consume, but they are not fundamentally changing the way they find adversaries in their network. Even if they employ the most sophisticated security analytics tools available, if they are sitting back and waiting for alerts, they are not hunting. HMM4 organizations, on the other hand, are actively trying new methods to find the threat actors in their systems. They try new ideas all the time, knowing that some won’t pan out but others will. They are inventive, curious, and agile, qualities you can’t get from a purely automated detection product. Although a good hunting platform can certainly give your team a boost, you can’t buy your way to HMM4. Bianco recommends HMM2 for CISOs looking to start hunting operations (2015). HMM2 describes organizations that are able to learn and apply procedures developed by others, and may make minor changes, but are not yet capable of creating wholly new procedures themselves. They routinely apply these procedures, if not on a strict schedule, then at least on a somewhat regular basis.
A couple recommendations I would make for organizations looking to implement hunting operations would be to monitor endpoint process creation, as well as searching for indicators of compromise. Many organizations look for logs to analyze but as Carvey describes in his Dell SecureWorks presentation, a malicious attacker can repurpose syslog so that logs aren’t giving proper information, and this would have to be detected by monitoring for these processes (Carvey, 2015). Organizations should look for endpoint processes that show artifacts or indicators that malicious activity is occurring in the network. Indicators, like endpoint process artifacts, can show lateral movements in internal networks. Web shells can be used to gain access to an infrastructure, by compromising a web server, and then moving to internal systems. Examples include is a Windows server running Apache and WordPress or by manipulating an IIS server. An attacker can also gain access with a web shell to an SQL server from a web server. The attacker can gain access to a web server, put a web shell on it, and with RDP access on both servers, the attacker can access the web shell in Internet Explorer by connecting it to localhost. Then they use the web shell to issue SQL injection commands, using xp_cmdshell and then create a user account on the SQL server. This can be found by looking through the browsing history, to see where the attacker was accessing localhost. In this case, the organization wouldn’t have event logs, because the attacker deleted the web shell after they were done, but there would be logs in the web server. Another file system indicator can be found on IIS servers with ASPX web shells, because the first time it is accessed, the .NET framework creates a page called the_name_of_the_web_shell.compile. In other words, the framework actually compiles it. These are file system artifacts that a hunter should look for when looking for advanced persistent threats, because attackers can come in, install a web shell, delete it after use, and repeat this process as much as they want, all the while going undetected in the network, because they actually created a legitimate login to the SQL or IIS server. Only if someone was actively looking for those indicators would they find out that web shells had been installed by malicious users. If an attacker crashes a web browser, it will create a session restore file, which, if the attacker doesn’t reinitiate the browser to delete that file, will remain on the system. Parsing through a compromised system after it has been taken offline will allow the forensics team to find these files and see what commands were issued through the web shell, as well as the username and password that the attacker used to access the SQL server, because the username and password would get stored in a config file. Carvey states that clusters of indicators, not individual artifacts, should be looked for, “because there are a lot of things that go on within an infrastructure that, if you look at them in isolation from everything else, could look like threat actor activity, because a lot of the stuff that we see threat actors doing is stuff that a normal admin might do” (2015).
Process creation monitoring is useful in live detection of attacks being carried out. This enables security professionals to see commands used by attackers, as they are being used, like checking the time of the remote system, checking to see if the task is completed, reissuing the task. Hunters should look to see if a process was created, when it was created and compare that to the hours of operation of that organization or the working hours of the person that normally uses that endpoint and other clusters of indicators like registry keys, passwords that were used, event logs, file systems, etc. Take, for example, the sticky keys attack. In the Windows registry, there is a key called image file execution options with spaces between all the words, that Microsoft left in place so that users can add debugging capabilities to binaries. An attacker can modify this registry key, via RDP access to the system, with the reg.exe command line utility. The attacker creates a subkey for one of the two accessibility tools, hc.exe or utilman.exe, and points the debugger value to cmd.exe. Even if all the passwords in the organization’s infrastructure are changed, and the attacker can still access the infrastructure, all they must do is RDP to that system, and when the login screen shows up, instead of inputting credentials, they just hit the shift key five times, and get a system level command prompt. Attackers use command line tools to do anything on a system. Once in, they can create users, change passwords, dump passwords, and anything else. The only way to detect this is to monitor for process creation and see that cmd.exe is being launched in places that it shouldn’t be, perhaps at times or on systems that should show no use, or on systems on which users should not be launching cmd.exe. Another suggestion I would make is to make use of shimcache and amcache. This allows systems administrators to see what has been run on a system and when and for how long. This can be started by running it through Python directly or by making a Windows EXE from the Python script, provided on its GitHub Page, https://github.com/mandiant/ShimCacheParser. ShimCache data should be collected and analyzed from all Windows endpoints in an organization, both clients and servers. Servers are particularly important, because they are “usually the number one initial entry point for breaches, especially internet-facing servers, or other servers and DMZs,” says David Sharpe, in his DerbyCon 2015 talk (Sharpe, 2015). Amcache replaced Shimcache, starting with Windows Server 2012 and Windows 8, and provides the same function as Shimcache, but has more useful fields for hunting, such as an SHA1 hash of the file, as well as more useful timestamp fields. Data from these caches should be stacked and analyzed for sequences of recon activity, net commands, pings, archivers, like RAR, being ran, and EXEs running out of abnormal locations on the disk. An example would be if an Amcache timeline were created, and EXEs were found, being run from the C:users location, this would be an abnormal location for EXEs to be run, as this does not normally occur.
I would also recommend mining server antivirus logs, because they are a consistent, high yield data source to hunt for intrusions, which is especially true for internet-facing assets. According to Sharpe, about 20% of all targeted intrusions have AV fire somewhere along the timeline (2015). If an intrusion attempt has progressed far enough along to where an AV product triggers, then that is helpful. At best, there will be a blocked intrusion, but there will still be an exploitable hole that needs to be addressed. The worst case scenario is that the intrusion is far along and AV picked up one tool in a long series of events that need to be addressed. Things to look for include web shells, AV detections while the file is under webroot or C:windows, any kind of backdoors and malware street names identified by intelligence sources and experience. This should be supplemented by custom host intrusion prevention systems detection, with HIPS rules targeting how malware tools work. Look for credential dumpers, like WCE, pwdump, gsecdump, fgdump, or Mimikatz.
Netstat data should be mined to find rogue listeners across all endpoints, especially servers on the network edges. The command netstat -nabo to pull the data and mine it. An example of an indicator of compromise could be if one TCP port has bound to it multiple process names and paths on a single system. This would be impossible on a normally-running system. In this case, intruder activity could be interleaved with legitimate SQL server activity. Netstat data output should be stacked for all internet-accessible servers by listening port, and see how many ports show up just once. This data should also be stacked by the full path to the process’ binary, and see how many paths show up just once. Additionally, all output should be preserved as a baseline, and all new listeners that appear over time, especially those across internet-facing systems, should be tracked (Sharpe, 2015).
There are many companies that offer proactive hunting services for fees, but I would recommend that an organization also have in-house hunters that are proactively seeking out cyberattacks. Outside consultation should be utilized in order to improve in-house hunting. In striving to be an organization with competent cybersecurity measures in place, the organization should collect very large amounts of data from across the enterprise and at all endpoints. All the suggestions I have made in this paper have involved compiling large amounts of data sets to find abnormalities in the operations of the organization. It is only with these data sets that we can analyze the data and find indicators of compromise.
Ashford, W. (2015, October 13). Cyber security innovation is crucial, says security evangelist. Retrieved December 15, 2017, from http://www.computerweekly.com/news/4500255332/Cyber-security-innovation-is-crucial-says-security-evangelist
Ashford, W. (2016, March). Hunters: a rare but essential breed of enterprise cyber defenders. Retrieved December 15, 2017, from http://www.computerweekly.com/feature/Hunters-a-rare-but-essential-breed-of-enterprise-cyber-defenders
Bianco, D. (2015, October 15). A Simple Hunting Maturity Model. Retrieved December 15, 2017, from http://detect-respond.blogspot.com/2015/10/a-simple-hunting-maturity-model.html
Carvey, H. (2015, July 25). BsidesCincy 2015 01 Lateral Movement Harlan Carvey. Retrieved December 15, 2017, from https://www.youtube.com/watch?v=dYoYMsJ5aIc
Lee, R. M. (2016, February). The Who, What, Where, When, Why and How of Effective Threat Hunting. Retrieved December 15, 2017, from https://www.sans.org/reading-room/whitepapers/analyst/who-what-where-when-effective-threat-hunting-36785
Rackspace. (2017, September 29). AGE OF THE CYBER HUNTER: HOW A NEW GENERATION OF THREATS CHANGED THE CYBERSECURITY PARADIGM. Retrieved December 15, 2017, from https://www.rackspace.com/sites/default/files/white-papers/age-of-the-cyber-hunter-white-paper_1.pdf
Rackspace. (2017). ENTERPRISE SECURITY TODAY – WHY SPEED MATTERS. Retrieved December 15, 2017, from http://go.rackspace.com/brand-deepdives26.html
Sharpe, D. (2015, September 28). Fix Me19 Intrusion Hunting for the Masses A Practical Guide David Sharpe. Retrieved December 15, 2017, from https://www.youtube.com/watch?t=1=MUUseTJp3jM