Step 1: Allocate physical or virtual systems for the analysis lab

A common approach to examining malicious software involves infecting a system with the malware specimen and then using the appropriate monitoring tools to observe how it behaves. This requires a laboratory system you can infect without affecting your production environment.

The most popular and flexible way to set up such a lab system involves virtualization software, which allows you to use a single physical computer for hosting multiple virtual systems, each running a potentially different operating system. Free virtualization software options include:

• VirtualBox
• VMware vSphere Hypervisor
• Microsoft Virtual Server

Running multiple virtual systems simultaneously on a single physical computer is useful for analyzing malware that seeks to interact with other systems, perhaps for leaking data, obtaining instructions from the attacker, or upgrading itself. Virtualization makes it easy to set up and use such systems without procuring numerous physical boxes.

Another useful feature of many virtualization tools is the ability to take instantaneous snapshots of the laboratory system. This way, you can record the state of the system before you infect it, and revert to the pristine environment with a click of a button at the end of your analysis.

If using virtualization software, install as much RAM into the physical system as you can, as the availability of memory is arguably the most important performance factor for virtualization tools. In addition, having a large hard drive will allow you to host many virtual machines, whose virtual file systems typically are stored as files on the physical system's hard drive.

Because malware may detect that it's running in a virtualized environment, some analysts prefer to rely on physical, rather than virtual, machines for implementing laboratory systems. Your old and unused PCs or servers can make excellent systems for your malware-analysis lab, which usually doesn't need high-performing CPUs or highly redundant hardware components.

To allow malware to reach its full potential in the lab, laboratory systems typically are networked with each other. This helps you observe the malicious program's network interactions. If using physical systems, you can connect them with each other using an inexpensive hub or a switch.

You must take precautions to isolate the malware-analysis lab from the production network, to mitigate the risk that a malicious program will escape. You can separate the laboratory network from production using a firewall. Better yet, don't connect laboratory and production networks at all, to avoid firewall configuration issues that might allow malware to bypass filtering restrictions.

If your laboratory network is strongly isolated, you can use removable media to bring tools and malware into the lab. Consider using write-once media, such as DVDs , to prevent malicious software from escaping the lab's confines by writing itself to a writable removable disk. A more convenient option is a USB key that includes a physical write-protect switch.

Some malware-analysis scenarios benefit from the lab being connected to the internet. Avoid using the production network for such connectivity. If possible, provision a separate, and usually inexpensive, internet connection, perhaps by dedicating a DSL or Cable Modem line to this purpose. Avoid keeping the lab connected to the internet all the time to minimize the chance of malware in your lab attacking someone else's system on the internet.

If virtualizing your lab, be sure to keep up with security patches released by the virtualization-software vendor. Such software may have vulnerabilities that could allow malware to escape from the virtual system you infected and onto the physical host. Furthermore, don't use the physical machine that's hosting your virtualized lab for any other purpose.

Before you're ready to infect your laboratory system with the malware specimen, you need to install and activate the appropriate monitoring tools. Free utilities that will let you observe how Windows malware interacts with its environment include:

• File system and registry monitoring: Process Monitor with ProcDOT offer a powerful way to observe how local processes read, write, or delete registry entries and files. These tools can help you understand how malware attempts to embed into the system upon infection.
• Process monitoring: Process Explorer and Process Hacker replace the built-in Windows Task Manager, helping you observe malicious processes, including local network ports they may attempt to open.
• Network monitoring: Wireshark is a popular network sniffer, which can observe laboratory network traffic for malicious communication attempts, such as DNS resolution requests, bot traffic, or downloads.
• Change detection: Regshot is a lightweight tool for comparing the system's state before and after the infection, to highlight the key changes malware made to the file system and the registry.

Behavioral monitoring tools can give you a sense for the key capabilities of malicious software. For further details about its characteristics, you may need to roll up your sleeves and perform some code analysis.

Examining the code that comprises the specimen helps uncover characteristics that may be difficult to obtain through behavioral analysis. In the case of a malicious executable, you rarely will have the luxury of access to the source code from which it was created. Fortunately, the following free tools can help you reverse compiled Windows executables:

• Disassembler and debugger: OllyDbg and IDA Pro Freeware can parse compiled Windows executables and, acting as disassemblers, display their code as assembly instructions. These tools also have debugging capabilities, which allow you to execute the most interesting parts of the malicious program slowly and under highly controlled conditions, so you can better understand the purpose of the code.
• Memory dumper: Scylla and OllyDumpEx help obtain protected code located in the lab system's memory and dump it to a file. This technique is particularly useful when analyzing packed executables, which are difficult to disassemble because they encode or encrypt their instructions, extracting them into RAM only during run-time.

To round off your malware-analysis toolkit, add to it some freely available online tools that may assist with the reverse engineering process. One category of such tools performs automated behavioral analysis of the executables you supply. These applications look similar at first glance, but use different technologies on the back end. Consider submitting your malware specimen to several of these sites; depending on the specimen, some sites will be more effective than others. Such tools include:

• Anubis
• EUREKA
• Malwr
• ThreatExpert

You can see a longer list of free automated malware analysis services that can examine compiled Windows executables.

Another set of potentially useful online tools provides details about websites that are suspected of hosting malicious code. Some of these tools examine the sites you specify in real time; others provide historical information. Consider submitting a suspicious URL to several of these sites, because each may offer a slightly different perspective on the website in question:

• Real-time threat assessment: WebInspector, and Wepawet
• Historical reputation data: URLVoid and MxToolbox

You can see a longer list of free on-line tools for looking up a potentially malicious website.

With your initial toolkit assembled, start experimenting in the lab with malware you come across on the web, in your e-mail box, on your systems, and so on. There are several "cheat sheets" that can help you in this process, including:

• Reverse-Engineering Malware Cheat Sheet
• Analyzing Malicious Documents Cheat Sheet

Begin analysis with the tools and approaches most familiar to you. Then, as you become more familiar with the inner workings of the malware specimen, venture out of your comfort zone to try other tools and techniques. The tools I've listed within each step operate virtually identically. Since they're all free, you should feel free to try them all. You'll find that one tool will work better than another, depending on the situation. And with time, patience, and practice, you will learn to turn malware inside out.

For additional tips and resources, see my article How to Get Started With Malware Analysis and check out the Reverse-Engineering Malware course I teach at SANS Institute