Some times ago I stumbled on a post at Offensive Computing where an user had some problems trying to figure out the encryption used by a malware (md5=36401cb9178232dde01b2788e8fc56f4).
The malware contains 2 files, located in the resource section. The files are encrypted, how to find out the encryption scheme? Well, using a debugger I would say. I didn’t start any debugger btw, deciding to take a look at the files using a resource editor. It’s a common thing to store files inside resource section. Here are the starting bytes of the first file:

Rot13_file

If you know which are the first common bytes in an exe file you should be able to figure out yourself which kind of encryption has been used. The presence of many 0×13 bytes is a nice hint, the file has been rot13h-ed. Just to make things clear, I used the word rot13h which is different from the well known rot13 encryption.

In an old blog entry I talked about a little ida plugin able to extract and analyze an hidden file; I slightly changed the plugin adding the possibility to un-rot13h the hidden file.

Rot13_dialog

Download the plugin from here