I find that more and more often binaries are being packed with exe protectors such as upx, aspack etc. I tried to follow a few tutorials on how to unpack them but the examples are often quite easy while my targets are not.
I am looking for good resources and any hints/tips on how to unpack targets.
Unpacking a generic wrapping packer or cryptor usually involves the following steps:
This is not difficult with simple packers but might be tricky with the more advanced ones. They may employ timing checks (rdtsc), exception-based control transfer, using debug registers for calculations etc. Using a VM or an emulator here usually helps against most of them.
There are many ways to do this. Sometimes the jump to OEP is obvious when it follows a chunk of looping code and there's nothing reasonable-looking after it. Or you may recognize the code at OEP if you're familiar with the entrypoints produced by different compilers. A couple other tricks:
if the packer saves the original registers before unpacking, set a hardware breakpoint on their location in the stack - this way you'll break right when they're restored before jumping to OEP.
if during tracing you can identify memory where the unpacked code is being written, set a page execution breakpoint on that memory range - it will trigger after the jump. IDA allows you to set such a breakpoint, and I think OllyDbg too.
set breakpoints on common APIs used by startup code, e.g. GetCommandLine or GetVersionEx. This won't get you the exact OEP, but you can usually go back the callstack and find it more or less easily.
If you're using IDA, you don't actually need to dump the file into a separate file - it's enough to take a memory snapshot that would copy the bytes from memory to the database so you can analyze them later. One thing to keep in mind here is that if the packer used dynamically allocated memory, you need to mark it as "loader" so it gets included in the snapshot. More here.
I'm not very familiar how it's done in Olly or other debugger, but AFAIK you need to use a tool like ImpREC on your dump and a copy of the process in memory.
It's somewhat simpler (IMO) in IDA. You just need to find the import table and rename the pointers according to the functions they are currently pointing to (this should be done while debugger is active). You can use either renimp.idc script or UUNP "manual reconstruct feature" (see here).
For finding import table there are two tricks I sometimes use:
follow some calls in the startup code at OEP to find external APIs and this should lead you to the import table. Usually the start and the end of the table is obvious.
during unpacking, set a breakpoint on GetProcAddress and see where the results are written. This however won't work with packers that use manual import resultion using the export directory. Putting a read BP on kernel32's export table might help here.
This is optional but it may be useful to remove the remains of the packer code that would only distract you. In IDA, you should also apply a compiler FLIRT signature if you recognize the compiler used.
I don't do this step as I rarely need to run the unpacked file but in general you usually need to fix up the PE header so that offsets to the section's code in file match those in the dump.
Now, there are many variations and tricks not covered by the above steps. For example, some packers don't fully resolve imports initially but put jumps to stubs that resolve import on first call and then patch it so it goes directly to the target next time. Then there is "stolen code" approach which makes it harder to find and recover OEP. Sometimes the packer runs a copy of itself and debugs it, so that you can't attach your own debugger to it (this can be solved by using emulator or a debugger that doesn't use debugging APIs like Intel PIN). Still, the outlined steps can cover quite a lot of what's out there.
I will conclude with the video that Elias made showing the process of unpacking the Lighty Compressor: https://www.hex-rays.com/video/bochs_video_2.html
Igor's answer is very good. However, the outlined techniques rely on the assumption that at some point the executable is unpacked in memory. This is not always true. Virtualization obfusactors compile the original binary into a custom instruction set when is executed by an simulator at runtime. If you encounter a binary obfuscated in this way you have no choice but to write a disassembler from the custom instruction set to an instruction set that you understand.
Blackstorm portal has a huge collection of Unpacking tutorials Blackstorm portal tutorials
Tuts4You has another large collection of unpacking tutorials Tuts4You
It took me a long time at first but over time unpacking got a lot easier, lots of patience and practice required though.
