Optimize a Makefile

Question

Since my project is getting bigger every day and I am just a starter in the wonderful world of makefiles, I need some help improving mine because, although it works (almost) as I wish, it really started looking like a mess. So it would be nice if someone could help me with it (and of course, advice is welcome).

Basically this is the structure of my C++ project:

myProject
| doc/ (nothing to do here)
| obj/ (where all *.o go)
| src/ (where I have all my *.h and *.cpp)
| tests/ (where all my tests are)

I have to say, in my Makefile I have some normal stuff, but also some really ugly stuff, so I hope you do not panic:

define \n
endef
EXECUTABLE = main
# compiler
CC = g++
CFLAGS = -g -std=gnu++0x -Wall -Wno-reorder -I. $(SYSTEMC_INCLUDE_DIRS)
LFLAGS = $(SYSTEMC_LIBRARY_DIRS)
FINAL = -o $(EXECUTABLE)
LIBS = -lsystemc-ams -lsystemc | c++filt
# directory names
SRCDIR = src
OBJDIR = obj
TSTDIR = tests
SOURCES := $(wildcard $(SRCDIR)/*.cpp)
INCLUDES := $(wildcard $(SRCDIR)/*.h)
TEST_SRC := $(wildcard $(TSTDIR)/*.cpp)
OBJECTS := $(SOURCES:$(SRCDIR)/%.cpp=$(OBJDIR)/%.o)
TESTS := $(TEST_SRC:$(TSTDIR)/%.cpp=$(TSTDIR)/%.o)
rm = rm -rf
all: $(EXECUTABLE)
check: testbenches run_test_script
debug: CC += -O0 -fno-inline
debug: all
main: createdir maincpp $(OBJECTS)
 $(CC) $(CFLAGS) $(LFLAGS) $(FINAL) $(OBJDIR)/[email protected] $(OBJECTS) $(LIBS)
TBS = $(basename $(TEST_SRC))
testbenches: createdir $(OBJECTS) $(TESTS)
 $(foreach tb, $(TBS), $(CC) $(CFLAGS) $(LFLAGS) -o $(tb).tst $(tb).o $(OBJECTS) $(LIBS) ${\n})
run_test_script:
 @cd $(TSTDIR); \
 ./run_tests.sh
createdir:
 @mkdir -p obj
maincpp:
 $(CC) $(CFLAGS) -c -o $(OBJDIR)/main.o main.cpp
$(TESTS): $(TSTDIR)/%.o : $(TSTDIR)/%.cpp
 $(CC) $(CFLAGS) -c -o $@ $<
$(OBJECTS): $(OBJDIR)/%.o : $(SRCDIR)/%.cpp
 $(CC) $(CFLAGS) -c -o $@ $<
.PHONY: clean
clean:
 $(rm) $(OBJDIR)
 $(rm) $(TSTDIR)/*.o
 $(rm) $(TSTDIR)/*.out
 $(rm) $(subst .cpp,.tst, $(TEST_SRC))
 $(rm) $(EXECUTABLE)

• Nobody moving away from SE: If it is not for training purposes I would advice you to use a Makefile generator like autotools, cmake, ...
• makeMonday: Well, when I started this project I chose to do Makefiles manually to learn, so in some way it is. Anyway, I'll take a look at Autotools, because at this point, I certainly need a more organized and less dependent (on me) build system.

Answer 1

You have many more .PHONY targets than just clean: all, check, debug, testbenches, run_test_script, createdir, maincpp.

The maincpp rule should be:

$(OBJDIR)/main.o: main.cpp
 $(CC) $(CFLAGS) -c -o $@ $<

The createdir action is @mkdir -p obj, but should be written as @mkdir -p $(OBJDIR).

main: createdir $(OBJECTS) has an improper prerequisite createdir. There is no guarantee that createdir will be made before $(OBJECTS). I would eliminate createdir altogether and write instead:

$(OBJECTS): $(OBJDIR)/%.o : $(SRCDIR)/%.cpp
 @mkdir -p $(@D)
 $(CC) $(CFLAGS) -c -o $@ $<

I would rewrite the main rule as:

$(EXECUTABLE): $(OBJDIR)/main.o $(OBJECTS)
 $(CC) $(CFLAGS) $(LFLAGS) -o $@ $+ $(LIBS)

The variable for the C++ compiler is typically $(CXX) instead of $(CC). (You might not even need to define it.)

---

All together, with a few similar cleanups for the tests...

EXECUTABLE = main
# compiler
CXX = g++
CFLAGS = -g -std=gnu++0x -Wall -Wno-reorder -I. $(SYSTEMC_INCLUDE_DIRS)
LFLAGS = $(SYSTEMC_LIBRARY_DIRS)
LIBS = -lsystemc-ams -lsystemc | c++filt
# directory names
SRCDIR = src
OBJDIR = obj
TSTDIR = tests
SOURCES := $(wildcard $(SRCDIR)/*.cpp)
INCLUDES := $(wildcard $(SRCDIR)/*.h)
OBJECTS := $(SOURCES:$(SRCDIR)/%.cpp=$(OBJDIR)/%.o)
TEST_SRC := $(wildcard $(TSTDIR)/*.cpp)
TESTS := $(TEST_SRC:$(TSTDIR)/%.cpp=$(TSTDIR)/%.o)
TEST_EXES = $(TESTS:$(TSTDIR)/%.o=%.tst)
rm = rm -rf
all: $(EXECUTABLE)
check: $(TEST_EXES)
 @cd $(TSTDIR); \
 ./run_tests.sh
debug: CXX += -O0 -fno-inline
debug: all
$(EXECUTABLE): $(OBJDIR)/main.o $(OBJECTS)
 $(CXX) $(CFLAGS) $(LFLAGS) -o $@ $+ $(LIBS)
$(TEST_EXES): %.tst : $(TSTDIR)/%.o
 $(CXX) $(CFLAGS) $(LFLAGS) -o $@ $< $(OBJECTS) $(LIBS)
$(TESTS): $(TSTDIR)/%.o : $(TSTDIR)/%.cpp
 $(CXX) $(CFLAGS) -c -o $@ $<
$(OBJDIR)/main.o: main.cpp
 @mkdir -p $(@D)
 $(CXX) $(CFLAGS) -c -o $@ $<
$(OBJECTS): $(OBJDIR)/%.o : $(SRCDIR)/%.cpp
 @mkdir -p $(@D)
 $(CXX) $(CFLAGS) -c -o $@ $<
clean:
 $(rm) $(OBJDIR)
 $(rm) $(TSTDIR)/*.out
 $(rm) $(TESTS) $(TEST_EXES)
 $(rm) $(EXECUTABLE)
.PHONY: all check debug clean

Answer 2

A couple of things I dislike about your set up.

1. Your low level make file in the top level directory.
2. You only have one object directory (so you can only have one type of build)
   I have four types of build debug/release/coverage/size(built with size optimization)
3. You use explicit commands where the makefile internal rules will work just as well.

1: At the top level your make file should just call the makefile in the source directory(s).

 # (you can have a target for all the commands you support)
 # (I use the following as my starting base)
 all:
 $(MAKE) -C src
 clean:
 $(MAKE) -C src clean
 debug:
 $(MAKE) -C src debug
 release: 
 $(MAKE) -C src release
 size:
 $(MAKE) -C src size
 test:
 $(MAKE) -c src test
 veryclean:
 $(MAKE) -C src veryclean
 install:
 $(MAKE) -C src install

Some other things I define in my make file:

 #
 # Basic block for building.
 # ?= define if not set on command line.
 ROOT ?= $(shell dirname `pwd`)
 BUILD ?= debug
 #
 # Set up SRC and OBJ directories.
 # Build debug/release/size and coverage into different directories.
 SRC_DIR = $(ROOT)/src
 OBJ_DIR = $(ROOT)/$(BUILD)
 #
 # Install by default done locally
 # but you do want to be able to install into the standard locations.
 PREFIX ?= $(ROOT)
 PREFIX_BIN ?= $(PREFIX)/bin
 PREFIX_INC ?= $(PREFIX)/include
 PREFIX_LIB ?= $(PREFIX)/lib

So

 sudo make install PREFIX=/usr

will install the code into standard locations in the OS (or /usr/local if you prefer).

2: Object files of different types can not be linked together.

You must use the exact same set of flags on every object file to guarantee they are binary compatible. So I build debug and release versions of the executables into different object directories. That way when linking there is no possibility of accidentally mixing objects of different types.

3: Default rules

The default rule for C++ code (source to object is)

%.o: %.cpp
 $(CXX) -c $^ $(CPPFLAGS) $(CXXFLAGS) 

The default rule for linking is

%: %.o
 $(CC) $(LDFLAGS) $^ $(LOADLIBES) $(LDLIBS)

I would adapt these rather than making your own set of variable names.

4: Commands

Standard conventions means that variables should be in all caps.

rm = rm -rf

I would change that too:

RM = rm -rf

Also defining the C compiler to g++ may not always do what you want.

CC = g++

I would prefer to re-define the C compiler as the C++ compiler and if need be then be explicit about the C++ compiler.

# CXX = Define if needed. Defaults to the correct system compiler.
CC = $(CXX)

5: COMPILER flags.

Here you have a fixed set.

CFLAGS = -g -std=gnu++0x -Wall -Wno-reorder -I. $(SYSTEMC_INCLUDE_DIRS)

These look fine but I would not explicitly set them I would append to the ones defined by the makefile system:

CFLAGS += -g -std=gnu++0x -Wall -Wno-reorder -I. $(SYSTEMC_INCLUDE_DIRS)
 ## ^^^^^ Add my flags onto the default ones.

Also your flags include all things all the time. I would divide this up to define the flags based on the type of build you are doing.

CFLAGS += $(CFLAGS_$(BUILD)) -std=gnu++0x -Wall -Wno-reorder -I. $(SYSTEMC_INCLUDE_DIRS)
CFLAGS_debug = -g
CFLAGS_release = -O3

The -Wall is a good starting point. But it is by no way All the warning flags (just a small subset). I personally use a few more:

-Wall -Wextra -Wstrict-aliasing -ansi -pedantic -Werror -Wunreachable-code

6: The last thing I do is encapsulate all my rules in a generic makefile.

Thus each project makefile only defines exactly what I need (and then includes the generic makefile). That way it is easy to see what I am actually building (and mistakes only need to be corrected in one place).

Example of Generic makefile

# My generic make file depends on this environment variable
THORSANVIL_ROOT = $(realpath ../)
# This is what I want to build.
# My tools support a couple of extensions
# app: An executable.
# slib: A shared lib
# a: A static lib
# dir: calls make -C <sub dir>
# head: A header only C++ library
TARGET = Serialize.slib
#
# Generic extension applied to all files. extensions.
LINK_LIBS = Json
UNITTEST_LINK_LIBS = Json
FILE_WARNING_FLAGS += -Wno-overloaded-virtual
#
# Specific extensions applied to all this source file
JsonSerilizeVardacTest_CXXFLAGS += -pedantic -Werror
#
# Now include the generic make file
# With all the rules I have built up.
include ${THORSANVIL_ROOT}/build/tools/Makefile

My generic build file can be found here. Have a look and take what you find useful.

7: Plug for things I do but thats because I am ecentric.

The one thing I hate about make fils is the long lines it prints when building. These are useless and just confuse the output.

g++ -c URILexer.cpp -o debug/URILexer.o -fPIC -Wall -Wextra -Wstrict-aliasing -pedantic -Werror -Wunreachable-code -Wno-long-long -Wno-unreachable-code -I/Users/myork/Repository/ThorWeb/build/include -isystem /Users/myork/Repository/ThorWeb/build/include3rd -DBOOST_FILESYSTEM_VERSION=3 -g -std=c++1y
g++ -c URIParser.cpp -o debug/URIParser.o -fPIC -Wall -Wextra -Wstrict-aliasing -pedantic -Werror -Wunreachable-code -Wno-long-long -Wno-unreachable-code -I/Users/myork/Repository/ThorWeb/build/include -isystem /Users/myork/Repository/ThorWeb/build/include3rd -DBOOST_FILESYSTEM_VERSION=3 -g -std=c++1y
g++ -c URITail.cpp -o debug/URITail.o -fPIC -Wall -Wextra -Wstrict-aliasing -pedantic -Werror -Wunreachable-code -Wno-long-long -Wno-unreachable-code -I/Users/myork/Repository/ThorWeb/build/include -isystem /Users/myork/Repository/ThorWeb/build/include3rd -DBOOST_FILESYSTEM_VERSION=3 -g -std=c++1y

So I make my makefile only print out the interesting stuff. If there are no errors all you get is the basics info you need (and a GREEN OK). You will get the full command line if there is an error.

Building debug
g++ -c URILexer.cpp -g OK
g++ -c URIParser.cpp -g OK
g++ -c URITail.cpp -g OK
g++ -c Fail.cpp -g
ERROR
g++ -c Fail.cpp -o debug/Fail.o -fPIC -Wall -Wextra -Wstrict-aliasing -pedantic -Werror -Wunreachable-code -Wno-long-long -Wno-unreachable-code -I/Users/myork/Repository/ThorWeb/build/include -isystem /Users/myork/Repository/ThorWeb/build/include3rd -DBOOST_FILESYSTEM_VERSION=3 -g -std=c++1y
========================================
< All the error messages.>

I like to keep my unit test separate from my source. So I put all my unit tests in a sub directory under the source. The makefile detects its presence and forces a build of the unit tests if you try and install the code. The unit tests also automatically do code coverage and fail if you don't get an average of 85% across all the files:

> make test
Building Objects for Testing and Coverage
flex URILexer.l OK
bison URIParser.y OK
g++ -c HostName.cpp -DCOVERAGE_ThorURI OK
g++ -c URIParserInterface.cpp -DCOVERAGE_ThorURI OK
g++ -c URILexer.cpp -DCOVERAGE_ThorURI OK
g++ -c URI.cpp -DCOVERAGE_ThorURI OK
g++ -c URILexer.lex.cpp -DCOVERAGE_ThorURI OK
g++ -c URINormalize.cpp -DCOVERAGE_ThorURI OK
g++ -c URIParser.tab.cpp -DCOVERAGE_ThorURI OK
g++ -c HostNamePublicSuffixData.cpp -DCOVERAGE_ThorURI OK
a - coverage/URIParser.tab.o
a - coverage/URILexer.lex.o
a - coverage/HostName.o
a - coverage/HostNamePublicSuffixData.o
a - coverage/URI.o
a - coverage/URILexer.o
a - coverage/URINormalize.o
a - coverage/URIParserInterface.o
Done
Building Unit Tests
Building coverage
g++ -c unittest.cpp -DCOVERAGE_ThorURI OK
g++ -c HostNameTest.cpp -DCOVERAGE_ThorURI OK
g++ -c URIParserTest.cpp -DCOVERAGE_ThorURI OK
g++ -c URITest.cpp -DCOVERAGE_ThorURI OK
g++ -c URILexerTest.cpp -DCOVERAGE_ThorURI OK
g++ -c URISortableTest.cpp -DCOVERAGE_ThorURI OK
g++ -o coverage/unittest.app -DCOVERAGE_ThorURI OK
 Done Building coverage/unittest
Done
rm coverage/unittest.o
Running Unit Tests
Running main() from gtest_main.cc
[==========] Running 53 tests from 5 test cases.
[----------] Global test environment set-up.
[----------] 2 tests from HostName
[ RUN ] HostName.StandardTest
[ OK ] HostName.StandardTest (0 ms)
[ RUN ] HostName.BasicCom
[ OK ] HostName.BasicCom (0 ms)
[----------] 2 tests from HostName (0 ms total)
[----------] 23 tests from URILexer
[ RUN ] URILexer.Schema
[ OK ] URILexer.Schema (0 ms)
[ RUN ] URILexer.SchemaValidChar
[ OK ] URILexer.SchemaValidChar (0 ms)
[ RUN ] URILexer.SchemaFail
[ OK ] URILexer.SchemaFail (0 ms)
[ RUN ] URILexer.RegName
[ OK ] URILexer.RegName (0 ms)
[ RUN ] URILexer.RegNameValidChar
[ OK ] URILexer.RegNameValidChar (0 ms)
[ RUN ] URILexer.RegNameFail
[ OK ] URILexer.RegNameFail (0 ms)
[ RUN ] URILexer.HostIPV4
[ OK ] URILexer.HostIPV4 (1 ms)
[ RUN ] URILexer.HostIPV4Fail
[ OK ] URILexer.HostIPV4Fail (0 ms)
[ RUN ] URILexer.HostUserName
[ OK ] URILexer.HostUserName (0 ms)
[ RUN ] URILexer.HostUserNameFail
[ OK ] URILexer.HostUserNameFail (0 ms)
[ RUN ] URILexer.IPv6Future
[ OK ] URILexer.IPv6Future (1 ms)
[ RUN ] URILexer.IPv6FutureFail
[ OK ] URILexer.IPv6FutureFail (0 ms)
[ RUN ] URILexer.IPv6addressPrefix
[ OK ] URILexer.IPv6addressPrefix (0 ms)
[ RUN ] URILexer.IPv6addressMain
[ OK ] URILexer.IPv6addressMain (1 ms)
[ RUN ] URILexer.WordSegmentInFullPath
[ OK ] URILexer.WordSegmentInFullPath (0 ms)
[ RUN ] URILexer.WordSegmentAbsolutePath
[ OK ] URILexer.WordSegmentAbsolutePath (0 ms)
[ RUN ] URILexer.WordSegmentRelativePath
[ OK ] URILexer.WordSegmentRelativePath (0 ms)
[ RUN ] URILexer.SegmentValidChar
[ OK ] URILexer.SegmentValidChar (0 ms)
[ RUN ] URILexer.SegmentInvalid
[ OK ] URILexer.SegmentInvalid (0 ms)
[ RUN ] URILexer.Query
[ OK ] URILexer.Query (0 ms)
[ RUN ] URILexer.QueryInvalid
[ OK ] URILexer.QueryInvalid (0 ms)
[ RUN ] URILexer.Fragment
[ OK ] URILexer.Fragment (0 ms)
[ RUN ] URILexer.FragmentInvalid
[ OK ] URILexer.FragmentInvalid (0 ms)
[----------] 23 tests from URILexer (3 ms total)
[----------] 8 tests from URIParser
[ RUN ] URIParser.SchemaHostEmptyPath
[ OK ] URIParser.SchemaHostEmptyPath (2 ms)
[ RUN ] URIParser.SchemaHostPortEmptyPath
[ OK ] URIParser.SchemaHostPortEmptyPath (1 ms)
[ RUN ] URIParser.SchemaHostRoot
[ OK ] URIParser.SchemaHostRoot (0 ms)
[ RUN ] URIParser.SchemaHostPath
[ OK ] URIParser.SchemaHostPath (1 ms)
[ RUN ] URIParser.SchemaHostFile
[ OK ] URIParser.SchemaHostFile (0 ms)
[ RUN ] URIParser.SchemaHostFileQuery
[ OK ] URIParser.SchemaHostFileQuery (0 ms)
[ RUN ] URIParser.SchemaHostFileFrag
[ OK ] URIParser.SchemaHostFileFrag (0 ms)
[ RUN ] URIParser.SchemaHostFileQueryFrag
[ OK ] URIParser.SchemaHostFileQueryFrag (0 ms)
[----------] 8 tests from URIParser (4 ms total)
[----------] 1 test from URISortable
[ RUN ] URISortable.comparable
[ OK ] URISortable.comparable (4 ms)
[----------] 1 test from URISortable (4 ms total)
[----------] 19 tests from URI
[ RUN ] URI.allURI
[ OK ] URI.allURI (0 ms)
[ RUN ] URI.InvalidURL
[ OK ] URI.InvalidURL (1 ms)
[ RUN ] URI.clone
[ OK ] URI.clone (0 ms)
[ RUN ] URI.mainParts
[ OK ] URI.mainParts (1 ms)
[ RUN ] URI.DomainParts
[ OK ] URI.DomainParts (0 ms)
[ RUN ] URI.Normalize
[ OK ] URI.Normalize (0 ms)
[ RUN ] URI.NormalizeLowerCase
[ OK ] URI.NormalizeLowerCase (0 ms)
[ RUN ] URI.NormalizeEncode
[ OK ] URI.NormalizeEncode (0 ms)
[ RUN ] URI.NormalizeDecodeALPHA
[ OK ] URI.NormalizeDecodeALPHA (1 ms)
[ RUN ] URI.NormalizeDecodeHyphen
[ OK ] URI.NormalizeDecodeHyphen (0 ms)
[ RUN ] URI.NormalizeDecodeHyphon
[ OK ] URI.NormalizeDecodeHyphon (0 ms)
[ RUN ] URI.NormalizeDecodePeriod
[ OK ] URI.NormalizeDecodePeriod (0 ms)
[ RUN ] URI.NormalizeDecodeTilda
[ OK ] URI.NormalizeDecodeTilda (0 ms)
[ RUN ] URI.NormalizeRemoveDefaultPort
[ OK ] URI.NormalizeRemoveDefaultPort (1 ms)
[ RUN ] URI.NormalizePath
[ OK ] URI.NormalizePath (0 ms)
[ RUN ] URI.NormalizePathEndsSlash
[ OK ] URI.NormalizePathEndsSlash (0 ms)
[ RUN ] URI.NormalizePathRelative
[ OK ] URI.NormalizePathRelative (0 ms)
[ RUN ] URI.NormalizeQueryCombine
[ OK ] URI.NormalizeQueryCombine (1 ms)
[ RUN ] URI.NormalizeQuerySort
[ OK ] URI.NormalizeQuerySort (0 ms)
[----------] 19 tests from URI (5 ms total)
[----------] Global test environment tear-down
[==========] 53 tests from 5 test cases ran. (16 ms total)
[ PASSED ] 53 tests.
/Applications/Xcode.app/Contents/Developer/usr/bin/make VERBOSE=NONE PREFIX=/Users/myork/Repository/ThorWeb/build CXXSTDVER=14 TARGET_MODE=coverage report_coverage COVERAGE=
Generating Coverage for HostName.cpp
Generating Coverage for HostNamePublicSuffixData.cpp
Generating Coverage for URI.cpp
Generating Coverage for URILexer.cpp
Generating Coverage for URINormalize.cpp
Generating Coverage for URIParserInterface.cpp
HostName.cpp 100%
HostNamePublicSuffixData.cpp 100%
URI.cpp 96%
URILexer.cpp 81%
URINormalize.cpp 0%
URIParserInterface.cpp 100%
OK Code Coverage Passed

Checking coverage for a particular file: Note: This line is printed out above as part of the test (I just add a specific file name).

> make VERBOSE=NONE PREFIX=/Users/myork/Repository/ThorWeb/build CXXSTDVER=14 TARGET_MODE=coverage report_coverage COVERAGE=URILexer.cpp
URILexer.cpp 81%
 -: 0:Source:URILexer.cpp
 -: 0:Graph:coverage/URILexer.gcno
 -: 0:Data:coverage/URILexer.gcda
 -: 0:Runs:1
 -: 0:Programs:1
 -: 1:
 -: 2:#include "URILexer.h"
 -: 3:#include <stdexcept>
 -: 4:
 -: 5:using namespace ThorsAnvil::Web;
 -: 6:
 178: 7:URILexer::URILexer(std::istream& input)
 -: 8: : URIBaseFlexLexer(&input, &std::cerr)
 -: 9: , tokenStart(0)
 -: 10: , tokenEnd(0)
 178: 11:{}
 -: 12:
 881: 13:int URILexer::yylex(URIParserInterface& pi)
 -: 14:{
 881: 15: tokenStart = tokenEnd;
 881: 16: int result = URIBaseFlexLexer::yylex();
 881: 17: tokenEnd = tokenStart + yyleng;
 881: 18: pi.parsedCharacters(yyleng);
 881: 19: return result;
 #####: 20:}
 -: 21:
 #####: 22:void URILexer::LexerError(const char* msg) {throw std::runtime_error(std::string("URI parsing error: ") + msg);}
 -: 23:
 -: 24:TokenMarker URILexer::tokenMark() const
 -: 25:{
 -: 26: // See URIUtils.h
 -: 27: //std::cerr << "Token Mark: " << tokenStart << " : " << tokenEnd << "\n";
 523: 28: return (tokenEnd << 16) | tokenStart;
 -: 29:}
 -: 30:
 -: 31:

8: Learn to use the tools for building makefiles.

I like my tools because they build stuff the way I want. BUT they are very brittle to changes in the environment. They need to be plugged into a automake or some other tool for creating makefiles so that they become more cross platform.

• makeMonday: Really interesting! I found it really useful. I have started reading about automake and autoconf. I'll give it a try, but your instructions look pretty neat. Thanks :)