Are intel hex files converted into a binary format before they are written to Atmega's flash?

Question

I have been going through some resources on bootloaders, programmers, etc (I'm very new to this specific subject) and how they work with the Atmega chips and Arduino board programming. My research may have been shallow but none of the online articles/blogs/forum posts that I have been through mentioned that the .hex file (in intel hex ASCII format) needs to be converted to any other kind of binary format. It did seem odd to me though that one would directly write ASCII plain text into an MCU's flash. Also, when using AVR Burn-o-Mat (a GUI front end for avrdude) I did notice a couple of times that the number of bytes being flashed was significantly less than the actual size of the hex file. Most recently I went through this library on github which converts intel hex into a binary format. So am I right in saying that the hex file needs to be converted to binary? So for instance, does avrdude basically do this before sending the binary stream to the bootloader (when uploading a sketch via the IDE)?

• Juraj: I use that library in a sketch which reads hex from SD card and writes it as binary to flash on Arduino Mega. github.com/JAndrassy/ArduinoOTA/blob/master/examples/Advanced/…
• the busybee: If you read the documentation of Avrdude, you will find that it is able to use different formats of the binary to flash, including binary. The command line argument includes the specifier.

Answer 1

Yes, the hex codes in the .hex file are read in by avrdude (or other programs) and written to the MCU in binary. A small microprocessor would not be able to read ASCII format commands.

See my answer about What happens when code is uploaded using the bootloader?

---

(Following on from some comments)

I did notice a couple of times that the number of bytes being flashed was significantly less than the actual size of the hex file

I think there is some confusion here, because of the different way numbers can be represented, or stored. Just for example, the number which we know as "twenty" could be written (in a file) as:

• 20 (decimal)
• 00020 (decimal with leading zeroes)
• 0014 (hex)
• 0024 (octal)
• 10100 (binary digits)

Now a program like Avrdude could read numbers in any of those formats, internally convert them to "binary" and send a single byte, being that binary number, to the MCU for placing into a program memory byte.

The .hex format, which the Arduino system produces as part of the compilation process, uses lines in this format:

Line format:
 :nnaaaatt(data)ss
 Where:
 : = a colon
 (All of below in hex format)
 nn = length of data part
 aaaa = address (eg. where to write data)
 tt = transaction type
 00 = data
 01 = end of file
 02 = extended segment address (changes high-order byte of the address)
 03 = start segment address *
 04 = linear address *
 05 = start linear address *
 (data) = variable length data
 ss = sumcheck
 * We don't use these

The above from comments in my file https://github.com/nickgammon/arduino_sketches/blob/master/Atmega_Hex_Uploader/File_Utils.ino

Example: Optiboot in all its glory (only 512 bytes):

:107E0000112484B714BE81FFF0D085E080938100F7
:107E100082E08093C00088E18093C10086E0809377
:107E2000C20080E18093C4008EE0C9D0259A86E02C
:107E300020E33CEF91E0309385002093840096BBD3
:107E4000B09BFECF1D9AA8958150A9F7CC24DD24C4
:107E500088248394B5E0AB2EA1E19A2EF3E0BF2EE7
:107E6000A2D0813461F49FD0082FAFD0023811F036
:107E7000013811F484E001C083E08DD089C08234E0
:107E800011F484E103C0853419F485E0A6D080C0E4
:107E9000853579F488D0E82EFF2485D0082F10E0AE
:107EA000102F00270E291F29000F111F8ED06801E7
:107EB0006FC0863521F484E090D080E0DECF843638
:107EC00009F040C070D06FD0082F6DD080E0C81688
:107ED00080E7D80618F4F601B7BEE895C0E0D1E017
:107EE00062D089930C17E1F7F0E0CF16F0E7DF06D8
:107EF00018F0F601B7BEE89568D007B600FCFDCFD4
:107F0000A601A0E0B1E02C9130E011968C91119780
:107F100090E0982F8827822B932B1296FA010C0160
:107F200087BEE89511244E5F5F4FF1E0A038BF0790
:107F300051F7F601A7BEE89507B600FCFDCF97BE46
:107F4000E89526C08437B1F42ED02DD0F82E2BD052
:107F50003CD0F601EF2C8F010F5F1F4F84911BD097
:107F6000EA94F801C1F70894C11CD11CFA94CF0C13
:107F7000D11C0EC0853739F428D08EE10CD085E9AC
:107F80000AD08FE07ACF813511F488E018D01DD067
:107F900080E101D065CF982F8091C00085FFFCCF94
:107FA0009093C60008958091C00087FFFCCF809118
:107FB000C00084FD01C0A8958091C6000895E0E648
:107FC000F0E098E1908380830895EDDF803219F02E
:107FD00088E0F5DFFFCF84E1DECF1F93182FE3DFCA
:107FE0001150E9F7F2DF1F91089580E0E8DFEE27F6
:047FF000FF270994CA
:027FFE00040479
:0400000300007E007B
:00000001FF

So, a line in the .hex file will start with a colon, then a length, then an address, then a transaction type, then some variable amount of data, and then a sumcheck. These are stored in the file in hexadecimal notation.

The very first thing we can deduce from this is that the file size will be over double the amount of memory that is programmed. For example, to program "AB" in hex, this would take two bytes in the .hex file ("A" followed by "B") but these would be converted into the equivalent one-byte "binary" number, which would end up in the MCU memory.

Then, because of the colon, the transaction type, the line length, and the sumcheck, the file will be somewhat longer still, than double the number of bytes to be programmed into the MCU. That is the file layout overhead.

• the busybee: "[...] would not be able [...]" is not correct as a general statement. The authors of the bootloader deliberately decided to use only binary data. ;-) You might also want to add that Avrdude is able to read multiple formats of the binary to flash, including binary.
• Nick Gammon: I wrote that in the context of the question "It did seem odd to me though that one would directly write ASCII plain text into an MCU's flash - the ASCII plain text file is not written into flash memory and then executed. Certainly a microprocessor could read ASCII text and interpret it under program control, but the ASCII text is not itself loaded into flash and executed.
• the busybee: I see, so it is the wording of "to read". You mean "to read for execution" and I understood "to read for programming the flash". As there might be more like me, would you mind to enhance the answer? -- Because it were a waste of space to store programs in ASCII coded hex, I did not think of this.
• Nick Gammon: I have attempted to explain it better.
• the busybee: :-D Wow. Yes, it's cool to explain Intelhex, and a good thing in the context. -- Please don't get it wrong, my humble issue targets just the "to read" in the second sentence, and it can be clarified satisfyingly with the content of your first comment.