Why do we need a format for binary executable files

Question

When binary files (i.e. executables) are saved they usually have a format (e.g. ELF or .out) where we have a header containing pointers to where data or code is stored inside the file. But why don't we store the binary files directly in the form of sequence of machine instructions.
Why do we need to store data separately from the code?
Secondly when the assembler creates a binary file is the file is among the above formats?

• user40980: You want to read about the application binary interface (and follow the related links) and reconsider your question.
• gnat: Sharing your research helps everyone. Tell us what you've tried and why it didn’t meet your needs. This demonstrates that you’ve taken the time to try to help yourself, it saves us from reiterating obvious answers, and most of all it helps you get a more specific and relevant answer. Also see How to Ask
• mouviciel: Storing the binary files directly in the form of sequence of machine instructions IS a format.
• Justin: A COM file is pretty much what you are describing. To execute a COM file the OS loads the contents of the file into memory at offset 0100h, and then jumps to that location. It turns out that this simple file format has severe limitations - complex executable formats include features such as relocation as solutions to those limitations.

Answer 1

There are several use cases for metadata like this, some being "necessary" and others being nice to have.

• At the very bare minimum (if the code isn't position-independent, which is still not the default), the loader needs to know at which address the code should go.
• On some platforms (including some in common use today, such as embedded micro controllers), there is separate memory for code and for data. The loader needs to be able to tell code from data to put the code into memory the CPU can execute from, and the data into memory the CPU can read and write.
• For binaries which aren't executable files but rather libraries, you need a symbol table to enable code using that library to call the right function.
• Even if there isn't physically separate memory for code and data, it's beneficial to set up memory such that the data is not executable and the code is not writable. Sure, you could let the program do that itself in initialization, but that's error prone, pointless code duplication for no good reason.
• Knowing the purpose of data sections can allow optimization. For example, if you have a separate .bss section for data that should be initialized to zero, a modern OS can lazily allocate zero'd pages to improve process creation time. If you have a .rodata section for read-only data, this data can go into separate ROM (if your computer has ROM), freeing up RAM and possibly bringing other benefits. When running multiple instances of a program, the data assumed read-only (including code) can be shared among those processes.

I'm sure there are more reasons, I'm not very familiar with the design decisions that go into a loader.

• Jules: Some other reasons: * It may be necessary to load a program or library at an address other than the one for which it was originally compiled (e.g. some other library is already using the same address; or perhaps for address-space layout randomization), and as you note most code is not position independent, so to do this we need to know the locations of all the addresses stored in the program so we can update them. * When paging, data sections can be moved into the page file, but code sections can be simply discarded and reloaded later. * Identifying dynamic libraries to import and...
• Jules: ... working out which symbols should be linked to which references. * Cryptographic code signing. * Execution environment options (e.g. in Windows there are flags for whether a process should be created with a console window, run in POSIX compatibility mode, etc...).

Answer 2

Because we want our computers to do more than just execute one fixed program.

Having a stream of opcodes in memory and endlessly stepping through it is fine if you have a single-purpose machine. But on a real computer, you want to achieve things such as

• have one binary call another and transfer control to it
• have one binary call another and get control back after it terminates
• multi-task between many programs and make it appear as if they were running simultaneously
• give one program monitoring or administration rights over another program to limit the things the other program can do
• and much, much more!

For all of this, you need executable software that treats other executable software as its input data, which means that you have to be able to express things such as "this code owns that memory address" or "those two processes are really instances of the same program and can share their static data segment".

A lot of the bytes in a binary file is, in fact, a string of machine code, but everything else has to be stored somehow as well, and a binary file format is just a spec saying how to do this.

• user7043: A stream of machine code is also data that can be handled by another program. Sure, it doesn't have a whole lot of metadata, and metadata is useful, but your examples don't seem like good examples of that. "Calling another binary" (I assume you mean creating a process?) is no different from starting up the process that "calls another binary". Context switching has nothing to do with that metadata, it works with any code whatsoever (e.g. JIT compiled code). Who owns what memory address is already managed by virtual memory, and it works for any allocated memory without such metadata.

Answer 3

Setting certain areas as data lets some architectures detect when trying to execute them (and then let the OS kill it).

Also there is an architecture design that splits code and data into separate address spaces so the file needs to make the difference between the 2 otherwise it wouldn't be able to access the data. The restrictions of the executable format comes from trying to be cross compatible between them (assuming the same instruction sets).