What does "magic tests" mean for the file command?

Question

I was reading about the file command and I came across something I don't quite understand:

file is designed to determine the kind of file being queried.... file accomplishes this by performing three sets of tests on the file in question:

• filesystem tests,
• magic tests,
• language tests

What are magic tests?

Answer 1

"magic" here refers to "magic numbers": a special value that's in a known place in a file that identifies its type. The file command has a database of these numbers and what type they correspond to. The library that goes with that database is called libmagic, and you can access that from your own programs.

They aren't necessarily "numbers" as we might think of them. For example, a PNG image file always starts with "\x89PNG\r\n\x1a\n", a Java class begins with the four bytes (in hexadecimal) CA FE BA BE, and an HTML file has "< html" somewhere near the start. It's just some small sequence of data that's known to be in a file of that type, usually very close to the start.

When people are defining file formats they often include one of these in it either deliberately or just as part of making the format fit together. file can use them afterwards. It also has other ways of actually looking at the contents of the file to guess what it is ("language tests").

• IMSoP: Note that originally, the "magic numbers" were, specifically, the first two bytes of an executable file, used by the kernel to load it in the appropriate way. #! is actually an example of this, because the kernel itself, on seeing those bytes, is supposed to invoke the command that follows.

Answer 2

That refers to the "magic bytes" which many file formats have at the beginning of a file which show what kind of file this is.

E.g. if a file starts with #! then it is considered a script.

• Antonios Sarikas: Who writes these "magic bytes" in the file?
• Hauke Laging: @AntoniosSarikas The application which creates the file. Or the user, manually, e.g. when creating a shell script with an editor.