Detect an executable file in java

Question

I have a use case where we are allowing users to upload files. Now in the back end java(controller which extracts file from http request and checks), i want to detect if user uploads any executable file. If he uploads, i have to discard that file. I have googled it but couldn't find a fine solution. Some people suggested verifying extension(.exe). But i am not sure how far it will filter exe files. I want to completely block executable files from uploading.

If any of you have come across this scenario or have solution about this, please let me know. I would be grateful to you.

I would be more happy if you can point me to any JAVA implementation or Java API or algorithm which cam do this job.

• Rajeev: Any other suggestions?

Answer 1

I suspect that, apart from the extension-checking method you have already mentioned, there will be no way to catch every possible case. Executable files are ultimately sequences of machine instructions which make them largely indistinguishable from any other data.

Despite this however, there are things you could look for in certain types of executable. For example:

• Windows uses the Portable Executable format, which should always start with the magic number 4d5a (ASCII characters MZ)
• ELF format executable used by Linux start with 7f454c46
• Java class files always begin with cafebabe (that's hex, not ASCII!).
• As far as I can see, Mach-O files used by Mac-OSX have the magic number feedface (hex again)

I suggest you create a FileInputStream or similar and read the first few bytes of the file, checking for these magic numbers. It doesn't detect any file which contains executable code, but it should stop files in these standard executable formats from being allowed, which I think is what you hoped for.

So for example:

public static boolean isExecutable(File file) {
 byte[] firstBytes = new byte[4];
 try {
 FileInputStream input = new FileInputStream(file);
 input.read(firstBytes);
 // Check for Windows executable
 if (firstBytes[0] == 0x4d && firstBytes[1] == 0x5a) {
 return true;
 }
 return false;
 }
 catch (Exception e) {
 e.printStackTrace();
 }
}

Also beware that it is possible to get a false positive, where you reject a file which was not executable. I don't know what type of file you are intending to have uploaded so you should consider how likely it is for this to happen.

• Rajeev: thanks for your suggestion. I have gone through wikis and found the similar kind of solutions. Below are the wikis i got : csn.ul.ie/~caolan/publink/winresdump/winresdump/doc/… I would really more happy, if you can guide me how to check the above three coniditions you mentioned in java
• devrobf: Yes, that confirms what I'd found: windows executables always start with ASCII MZ (or hex 4d5a). I've updated my answer to include Mac OSX as well
• devrobf: I've updated with a code sample for Windows. I'll leave it to you to work out how to test the others. Hope that helps.
• Rajeev: In my case, i don't have restriction like users are allowed to upload certain formats. He can upload any type. As a precautionary measures, we want to stop them uploading executable ones. If you have no proble, can you give sample program as i am new to java...
• devrobf: @shareef The links are provided in my answer. I simply looked up each format in turn. They're just the Wikipedia articles for each format.

Answer 2

To complete the response of devrobf: Each executable file (by which I mean that the file contains machine instructions) can be identified by the magic number contained in the file's metadata. The magic number is identified by it's size (in byte) and by it's offset (which can be different depending on the type of file). You can find a database that contains this information HERE.

For example EXE file :

Extension : EXE 
Signature : 4D 5A
Description : Windows|DOS executable file
MZ (ASCII) 
Sizet : 2 Bytes
Offset: 0 Bytes

As you will surely understand, doing a check only on the extension does not make it possible to determine with certainty what kind of executable file. As the proposed Cratylus. Why? because the following example:

touch notAnExecutableWithExtensionExe.exe

This command just create file with as extension "exe", but it's just file data.

Implementation in Java to make a correct check of any kind of file :

public enum ExecutableSignatures{
 WINDOWS_EXE("Windows|DOS executable file", (byte) 0x00, (byte) 0x02, 
 new byte[]{(byte)0x4d, (byte)0x5a}),
 JAVA_BYTECODE("Java Bytecode", (byte) 0x00, (byte) 0x04, 
 new byte[]{(byte)0xca, (byte)0xfe, (byte)0xba, (byte)0xbe});
 /* Here more enumeration */
private String description;
private byte offset;
private byte size;
private byte[] magicNumber;
private ExecutableSignatures(String description, byte offset, byte size, byte [] magicNumber){
 this.description = description;
 this.offset = offset;
 this.size = size;
 this.magicNumber = magicNumber;
}
public String getDescription(){
 return this.description;
}
public byte getOffset(){
 return this.offset;
}
public byte getSize(){
 return this.size;
}
public byte[] getMagicNumbers(){
 return this.magicNumber;
}

After you can create a method to make this check by using apache librairies see HERE see @Filters - MagicNumberFilter. This constructor can take 2 paramaters; the magicNumbers(byte array) and the offset(byte).

 /**
 * Perform a check of what kind of executable is by checking the signature 
 * of file.
 * If it's an executable that is enumerate then the attributes 
 * magicNumber and executableDescription are updated with their corresponding 
 * values.
 * @return true if is an executable supported by the program otherwise false
 */
 public boolean isExecutableFile(){
 MagicNumberFileFilter mnff = null;
 for(ExecutableSignatures es : EnumSet.allOf(ExecutableSignatures.class)){
 mnff = new 
 MagicNumberFileFilter(es.getMagicNumbers(), es.getOffset());
 if(mnff.accept(this.file)){
 this.magicNumber = es.getMagicNumbers();
 this.executableDescription = es.getDescription();
 return true;
 }
 }
 return false;
 }

Answer 3

Windows executable always starts with MZ magic number. Probably you could check for this.

• Rajeev: I have gone through wikis and found below two wikis: delphidabbler.com/articles?article=8&part=1 stackoverflow.com/questions/2863683/… But not so much clear. They have suggested for cheking PE..But need more info on that like how to check and implement

Answer 4

As far as I have seen, the most usual approach is to verify the extension. For example I have noticed that mail clients usually accept to send an executable if it is renamed e.g. to zip or some other extension.
I believe that this seems adequate since the security problem is if the user accidentally runs the executable. By renaming the file to an unknown/different extension the user can not accidentally do that and so the danger is somehow "mitigated"
Otherwise coming up with a way to look into the file contents to determine if you actually have an executable, I don't know how feasible/portable/reliable this is

Answer 5

Have a look here:

Is there a good way to determine if a file is executable in Java

Seems like this command may help: java.io.File.canExecute()

• devrobf: Unfortunately, that only determines whether or not a file has permission to execute - not whether it actually contains executable code
• Rajeev: As jazzbassrob said, this won't help. It can't detect executable data is there in file or not
• Rajeev: i have already gone through the link you mentioned. It is not related to question i am asking exactly....

Answer 6

Be aware that windows executable are not only .exe files so checking the extension will not be enough

If you want something advanced and difficult to fool, you can use a third-party tool like File for Windows it is a popular command line tool that was ported from Linux.

for example, if you want to check some file program.exe

C:\file -b "program.exe"

The result will be something like

PE32 executable for MS Windows <GUI> Intel

You can run this tool from a Java program using Runtime.getRuntime().exec()

See this question to learn how to run a command line program and get the output in Java

You can also check Apache Tika to get the file type from its content

• Rajeev: How can i do this in java?
• Rajeev: What are the dependency i need to add... It will be more helpful to everyone if u can provide sample program which detects executable files? Also thankful to you if you can throw light on negatives of using this method like edge cases for which it can miss or it can treat normal file as executable one.
• iTech: You do not need to add anything in Java (e.g. jars), if you are able to run the command from cmd you can just call it directly from Java using getRuntime().exec()
• Rajeev: i am adding it in webaapplication. So i would prefer to run as part of request response cycle. I can't execute on commandline. Mine is a big webapplication which serves millions of users.
• Rajeev: k..i will check. Thanks for ur response