Compiling with -flto and -nostartfiles results in no program at all

Question

Towards developing my own bootloader I've stumbled over an issue with avr-gcc. I am trying to compile my code with the compiler flags

-flto #to enable link time optimization
-nostartfiles #to omit the interrupt vector table

My bootloader consists of multiple files which are used depending of compile flags. All the sources are being compiled and linked while the code path only leads to some portions of them. So I wanted to get rid of the dead code and -flto (being a newer development) creates a much smaller machine code size than -ffunction-sections -fdata-sections with -Wl,--gc-sections -Wl,--print-gc-sections.

But when it comes to further reducing the machine code size by removing the interrupt vector table using -nostartfiles, the complete code gets optimized away leaving me with a zero byte output.

Has anyone experienced the same issue with that compiler flag combination?

• Maximilian Gerhardt: I have experienced that I couldn't compile an Arduino sketch anymore with LTO because the optmization was so aggressive that it optimized out certain ISRs of hardware peripherals. Maybe GCC thinks that the initial bootup function (at 0x0000 or wherever) is never called and kills everything? Maybe try sprinkling __attribute__ ((used)) over some functions and see what happens? (gcc.gnu.org/wiki/LinkTimeOptimizationFAQ)
• Kwasmich: This is it! It now produces output. But it is not smaller than the --gc-sections version any more. I suppose it was removing some parts already that were supposed to stay. Thank you!
• Maximilian Gerhardt: Great to hear! I think I'd be great if you'd answer the question yourself and include your specific critical piece of code that made it work.

Answer 1

As @MaximilanGerhardt wrote in his comment. LTO was optimizing away too much of the code. Usual programs start with the interrupt vector table that in case of a regular reset or startup point to the main function. That is why the compiler knows that main has an entry point and is used. The actual program starts at the table.

Removing the interrupt vector table with -nostartfiles causes the compiler not to know that main is used at all and optimizes it away.

To give the compiler a hint __attribute__((used)) has to be placed at all methods that have to stay and there is no code path leading to them otherwise.

• Chris Stratton: You've sort of stumbled into something that works, but it's not really the right answer. What you should be doing is using a linker script (traditionally .ld file) which puts your startup code (preferably a wrapper not main() itself) at the correct address, with a "keep" attribute. In the case of the AVR the "vectors" are really jump instructions, so if you only want to support the reset one you can probably just start your code there; in other architectures the vectors are pointers, so you'd need the address of your startup code followed by code.
• Chris Stratton: Also note that without startup code, initialized RAM variables won't work, as copying that initialization from flash to RAM is done by startup code; however PROGMEM type values will.