What are good ways to organize input files (Makefiles, SConstruct, CMakeLists.txt, etc.) to build automation software?

Question

One thing I like to do with my code is make sure that it's refactored into manageable pieces. However, when it comes to building the software, I find that whatever build automation software I end up using (lately it's been GNU Make or SCons) ends up becoming a complete mess. The input files look like long scripts that seem to defy easy refactoring. I'd like to be able to refactor them in some way, but the concept of "function" doesn't quite behave the same way in some build automation software as it does in a programming language, so I find it difficult to write manageable Makefiles or SConscript files for any projects that are moderately complicated.

Does anyone have any advice on writing manageable input files for build automation software? Software-agnostic advice would be best, but even advice about a specific build automation tool would be helpful, particularly Make or SCons, since that's what I've been using for projects.

Edit: As Thorbjørn has pointed out, I should add some context and usage examples. I'm finishing up my PhD in chemical engineering and I do research in computational science. (I'm a pro tem mod over at SciComp.SE, for those of you who visit.) My projects typically involve a mix of compiled languages (C, C++, Fortran) that do some of the heavy lifting, scripting languages (Python, Perl) for prototyping, and occasionally, domain-specific languages for prototyping or technical purposes.

I've added two examples below, roughly in the 250-line range. The problem for me is generally a lack of modularity. Some of these projects can be organized into modular units, and it'd be nice to abstract the parts of the build along those lines to make it easier for me and future maintainers to keep track of. Breaking each script up into multiple files has been one solution I've been toying around with in my head.

The second example is particularly important, because I'm going to have to a large number of files to it soon.

Here's what a 265-line Makefile might look like for me, taken from an actual project, and organized as best I can:

#!/usr/bin/make
#Directory containing DAEPACK library folder
daepack_root = .
library = $(daepack_root)/lib
wrappers = $(daepack_root)/Wrappers/DSL48S
c_headers = parser.h problemSizes.h
f77_headers=problemSizes.f commonParam.f
f90_headers=problemSizes.f commonParam.f90
includes = -I. -Iinclude -I/usr/include/glib-2.0 \
 -I/usr/lib/glib-2.0/include -I/usr/include/libxml2 \
 -I/usr/include/libgdome -I/usr/include/gtest/
#Fortran 77 environment variables
f77=gfortran
fflags=-ggdb -cpp -fno-second-underscore --coverage -falign-commons \
 -mcmodel=large -fbacktrace -pg 
flibs=
#Fortran 90 environment variables
f90=gfortran
f90flags=-ggdb -cpp -fno-second-underscore --coverage -falign-commons \
 -mcmodel=large -fbacktrace -pg 
f90libs=
#C environment flags
cc=gcc
cflags=-ggdb --coverage $(includes) -mcmodel=large 
clibs=
#Libraries for linking
libs=-L$(library) -ldaepack_sparse -lblas -llapack -ldl -lg2c \
 -lgdome -lxml2 -lgtest -lcunit -lcholmod -lamd -lcolamd -lccolamd \
 -lmetis -lspqr -lm -lblas -llapack -lstdc++ -lpcre
#Object files
objs=main.o $(dsl48sObjs) $(gdxObjs)
gdxObjs = gdxf9def.o gdxf9glu.o gamsglobals_mod.o 
commonObjs=libdsl48s_model.sl cklib.o parser.o $(gdxObjs)
originalModelObjs=originalModel.o dsl48sChemkinModule.o $(commonObjs)
cspSlowModelObjs=cspSlowModel.o dsl48sChemkinModuleSlow.o cspModule.o \
 $(commonObjs)
orthoProjModelObjs=orthoProjModel.o dsl48sChemkinModuleOrthoProj.o \
 orthoProjModule.o basisModule.o spqrUtility.o $(commonObjs)
#Shell environment variable definitions for FUnit
FCFLAGS := $(f90flags)
LDFLAGS := libdsl48s_model.sl cklib.o gdxf9glu.o parser.o spqrUtility.o \
 $(libs)
misc=*table *size.f 
output=*.out
#Ftncheck flags for static analysis of Fortran 77 code
ftnchekflags= -declare -include=. -library -style=block-if,distinct-do,do-enddo,end-name,goto,labeled-stmt,structured-end
all: ckinterp.exe parserTest.exe originalModel.exe cspSlowModel.exe \
 orthoProjModel.exe spqrUtilityTest.exe
#Check code style with lexical analyzer
 @echo Checking program style...
 ftnchek $(ftnchekflags) rhs.f
 ftnchek $(ftnchekflags) resorig.f
 ftnchek $(ftnchekflags) res.f
# ftnchek $(ftnchekflags) cklib.f
# ftnchek $(ftnchekflags) ckinterp.f
#Set up baseline coverage data file
 @echo Set up baseline coverage data file
 lcov -c -i -d . -o conpDSL48Sbase.info
#Run unit test on cspModule.f90
 @echo Running unit tests on cspModule.f90...
 funit cspModule
#Generate test coverage data for cspModule.f90
 @echo Generating test coverage data from cspModule.f90 tests...
 lcov -c -d . -o conpDSL48ScspTest.info
#Run unit test on orthoProjModule.f90
 @echo Running unit tests on orthoProjModule.f90...
 funit orthoProjModule
#Generate test coverage data for orthoProjModule.f90
 @echo Generating test coverage data from orthoProjModule.f90 tests...
 lcov -c -d . -o conpDSL48SgenProjTest.info
#Run unit tests on the parser C library
 @echo Running unit tests on parser in C...
 -G_SLICE=always-malloc G_DEBUG=gc-friendly valgrind -v --tool=memcheck \
 --leak-check=full --show-reachable=yes --leak-resolution=high \
 --num-callers=20 --log-file=parserTest.vgdump \
 ./parserTest.exe > parserTest.log
#Generate test coverage data for the parser wrapper C library
 @echo Generating test coverage data for the parser in C...
 lcov -c -d . -o conpDSL48SparserTest.info
#Run unit tests on the SparseQR C library
 @echo Running unit tests on SparseQR library in C...
 ./spqrUtilityTest.exe
#Generate test coverage data for the SparseQR C library
 @echo Generating test coverage data for the SparseQR C library...
 lcov -c -d . -o conpDSL48SsparseTest.info
#Run unit test on basisModule.f90
 @echo Running unit tests on basisModule.f90...
 funit basisModule
#Generate test coverage data for basisModule.f90
 @echo Generating test coverage data from basisModule.f90 tests...
 lcov -c -d . -o conpDSL48SbasisMod.info
#Combine test coverage data
 @echo Combine baseline and test coverage data...
 lcov -a conpDSL48Sbase.info \
 -a conpDSL48ScspTest.info \
 -a conpDSL48SgenProjTest.info \
 -a conpDSL48SbasisMod.info \
 -a conpDSL48SparserTest.info \
 -a conpDSL48SsparseTest.info \
 -o conpDSL48Stotal.info
#Post-process to remove coverage statistics from automatically 
#generated source code.
 @echo Removing coverage statistics for automatically generated source...
 lcov -r conpDSL48Stotal.info basisModule_fun.f90 \
 ckinterp.f cklib.f cspModule_fun.f90 davisSkodjeAd.f90 \
 davisSkodjeJac.f90 davisSkodjeRes.f90 davisSkodjeRhs.f90 \
 davisSkodjeSp.f90 gdxf9def.f90 gdxf9glu.c orthoProjModule_fun.f90 \
 jac.f jacorig.f resad.f resadp.f resorigad.f resorigadp.f ressp.f \
 resorigsp.f senrhs.f senrhsorig.f TestRunner.f90 \
 -o conpDSL48Stotal.info
#Generate HTML report of coverage data
 @echo Generate HTML report of coverage data...
 genhtml conpDSL48Stotal.info
 @echo Open "index.html" in browser for coverage results!
originalModel.exe: $(originalModelObjs) $(f90_headers) $(f77_headers) \
 $(c_headers)
 $(f90) $(f90flags) -o originalModel.exe $(originalModelObjs) $(libs)
originalModel.o: dsl48sChemkinModule.o $(commonObjs) $(f77_headers) \
 $(f90_headers) $(c_headers)
 $(f90) $(f90flags) -c -o originalModel.o originalModel.f90
cspSlowModel.exe: $(cspSlowModelObjs) $(f90_headers) $(f77_headers) \
 $(c_headers)
 $(f90) $(f90flags) -o cspSlowModel.exe $(cspSlowModelObjs) $(libs)
cspSlowModel.o: dsl48sChemkinModuleSlow.o cspModule.o $(commonObjs) \
 $(c_headers) $(f77_headers)
 $(f90) $(f90flags) -c -o cspSlowModel.o cspSlowModel.f90
orthoProjModel.exe: $(orthoProjModelObjs) $(f90_headers) $(f77_headers) \
 $(c_headers) resOrthoFast.o
 $(f90) $(f90flags) -o orthoProjModel.exe $(orthoProjModelObjs) \
 resOrthoFast.o $(libs)
orthoProjModel.o: dsl48sChemkinModuleOrthoProj.o orthoProjModule.o $(commonObjs) \
 $(c_headers) $(f90_headers) $(f77_headers) resOrthoFast.o basisModule.o
 $(f90) $(f90flags) -c -o orthoProjModel.o orthoProjModel.f90
dsl48sChemkinModule.o: dsl48sChemkinModule.f90 cklib.o problemSizes.h \
 parser.o $(c_headers) $(f90_headers)
 $(f90) $(f90flags) -c -o dsl48sChemkinModule.o dsl48sChemkinModule.f90 
dsl48sChemkinModuleSlow.o: dsl48sChemkinModuleSlow.f90 cspModule.o cklib.o \
 problemSizes.h parser.o $(c_headers) $(f90_headers)
 $(f90) $(f90flags) -c -o dsl48sChemkinModuleSlow.o \
 dsl48sChemkinModuleSlow.f90
dsl48sChemkinModuleOrthoProj.o: dsl48sChemkinModuleOrthoProj.f90 \
 orthoProjModule.o basisModule.o cklib.o problemSizes.h \
 parser.o $(c_headers) $(f90_headers)
 $(f90) $(f90flags) -c -o dsl48sChemkinModuleOrthoProj.o \
 dsl48sChemkinModuleOrthoProj.f90
basisModule.o: basisModule.f90 cklib.o spqrUtility.o commonParam.f90
 $(f90) $(f90flags) -c -o basisModule.o basisModule.f90
spqrUtility.o: spqrUtility.h spqrUtility.c
 $(cc) $(cflags) -c -o spqrUtility.o spqrUtility.c
spqrUtilityTest.exe: spqrUtilityTest.o spqrUtility.o
 $(cc) $(cflags) -o spqrUtilityTest.exe spqrUtilityTest.o \
 spqrUtility.o $(libs)
spqrUtilityTest.o: spqrUtilityTest.c spqrUtility.o
 $(cc) $(cflags) -c -o spqrUtilityTest.o spqrUtilityTest.c
cklib.o: cklib.f ckstrt.f
 $(f77) $(fflags) -c -o cklib.o cklib.f
ckinterp.exe: ckinterp.o
 $(f77) $(fflags) -o ckinterp.exe ckinterp.o
ckinterp.o: ckinterp.f
 $(f77) $(fflags) -c -o ckinterp.o ckinterp.f
#Recursive makefile inherited from previous graduate students
libdsl48s_model.sl: $(f77_headers) cklibDAEPACK.f
 cp $(wrappers)/makefile model.mk
 make -f model.mk
resOrthoFast.o: libdsl48s_model.sl
 $(f90) $(f90flags) -c -o resOrthoFast.o resOrthoFast.f90
problemSizes.f: problemSizes.fpp problemSizes.h
 cpp problemSizes.fpp problemSizes.f
 perl -p -i.bak -we 's/# /! /;' problemSizes.f
commonParam.f90: commonParam.f
 perl -p -i.bak -we 's/^#/!/;' commonParam.f
 echo "commonParam t f t fpp" | pref77tof90
 echo "commonParam /" | f77tof90
 perl -p -i.bak -we 's/integer a/!integer a/;' commonParam.f
 perl -p -i.bak -we 's/END //;' commonParam.f90
commonParam.f: commonParam.fpp problemSizes.h
 cpp commonParam.fpp commonParam.f
 perl -p -i.bak -we 's/^#/!/;' commonParam.f
cspModule.o: cspModule.f90
 $(f90) $(f90flags) -c -o cspModule.o cspModule.f90
orthoProjModule.o: gamsglobals_mod.o gdxf9def.o gdxf9glu.o orthoProjModule.f90 \
 formatLabels.f90
 $(f90) $(f90flags) -c -o orthoProjModule.o orthoProjModule.f90
gdxf9def.o: gdxf9def.f90
 $(f90) $(f90flags) -c -o gdxf9def.o gdxf9def.f90
gdxf9glu.o: gdxf9glu.c gdxf9def.o
#64-bit version of wrappers (with underscores)
 $(cc) $(cflags) -DCIA_LEX -DAPIWRAP_LCASE_DECOR -c -o \
 gdxf9glu.o gdxf9glu.c
#64-bit version of wrappers (without underscores, for C interoperability)
# $(cc) $(cflags) -DCIA_LEX -DAPIWRAP_LCASE_NODECOR -c gdxf9glu.c
#32-bit version of wrappers
# $(cc) $(cflags) -DAPIWRAP_LCASE_DECOR -c gdxf9glu.c -Iinclude
gamsglobals_mod.o: gamsglobals_mod.f90 gdxf9def.o gdxf9glu.o
 $(f90) $(f90flags) -c gamsglobals_mod.f90
parser.o: parser.c $(c_headers)
 $(cc) $(cflags) -c -o parser.o parser.c 
parserTest.exe: parserTest.o parser.o
 $(cc) $(cflags) -o parserTest.exe parser.o \
 parserTest.o $(libs)
parserTest.o: parserTest.cpp parser.o
 $(cc) $(cflags) -c -o parserTest.o parserTest.cpp
clean:
 -rm *.bak
 -rm *.f77
 -rm *.log
 -rm commonParam.f90
 -rm problemSizes.f
 -rm commonParam.f
 -make clean -f model.mk
 -rm model.mk
 -rm *.o
 -rm *.mod
 -rm $(misc)
 -rm *.exe
 -funit --clean
 -rm *.gcno
 -rm *.gcda
 -rm *.info
 -rm *.png
 -rm *.html
 -rm *.css
 -rm -rf html
 -rm *.pyc
 -rm *.lst

Here's a 245-line SConstruct file that I'm currently trying to organize for a project that's roughly as complex:

## \file SConstruct
# \brief Compiles the library and compiles tests.
#
import SCons
## \brief Build up directory names of each COIN library from package names
# and versions.
#
## Overall SCons environment
#
env = Environment();
flags = []
## Compile using debug versions?
#
debug = True
debugString = '-debug'
debugFlags = ['-ggdb']
dynamicLinkFlag = '-Wl,-rpath,'
if debug:
 flags += debugFlags
## Compile Google Test from scratch.
#
GTestVersion = '1.6.0'
GTestStem = 'gtest-' + GTestVersion
GTestBuildIncDir = [GTestStem,
 GTestStem + '/include',
 ]
GTestAllLib = env.Library('lib/libgtest.a', 'gtest-1.6.0/src/gtest-all.cc',
 CPPPATH = GTestBuildIncDir,
 CXXFLAGS = flags)
GTestMainLib = env.Library('lib/libgtest_main.a',
 'gtest-1.6.0/src/gtest_main.cc',
 CPPPATH = GTestBuildIncDir,
 CXXFLAGS = flags)
GTestIncDir = GTestStem + '/include/gtest'
GTestLibDir = 'lib'
GTestLibFlags = ['gtest', 'gtest_main', 'pthread']
## Armadillo matrix library
#
ArmadilloLibFlags = ['armadillo'];
## Quick reminder of SCons flags:
# CPPPATH = path of headers (include directories)
# LIBPATH = path of libraries
# LIBS = flags of libraries
# CXXFLAGS = C++ compilation flags
#
## Locations of libraries installed on system in standard locations
#
StdIncDir = '/usr/include'
StdLibDir = '/usr/lib'
## Configuration information for COIN libraries
#
CoinUtilsVersion = '2.6.4'
ClpVersion = '1.12.0'
OsiVersion = '0.103.0'
CbcVersion = '2.5.0'
## Some standard directory locations of COIN libraries, with slashes added for
# for convenience.
#
CoinLibLocation = '/usr/local/COIN/'
StdCoinIncDir = '/include/coin'
StdCoinLibDir = '/lib'
CoinUtilsStem = 'CoinUtils-' + CoinUtilsVersion
ClpStem = 'Clp-' + ClpVersion
OsiStem = 'Osi-' + OsiVersion
CbcStem = 'Cbc-' + CbcVersion
if debug:
 CoinUtilsStem += debugString
 CbcStem += debugString
 ClpStem += debugString
 OsiStem += debugString
## Build up include directory names for COIN projects from constituent parts.
#
CoinUtilsIncDir = CoinLibLocation + CoinUtilsStem + StdCoinIncDir
ClpIncDir = CoinLibLocation + ClpStem + StdCoinIncDir
OsiIncDir = CoinLibLocation + OsiStem + StdCoinIncDir
CbcIncDir = CoinLibLocation + CbcStem + StdCoinIncDir
## Build up library names from COIN projects from constituent parts
#
CoinUtilsLibDir = CoinLibLocation + CoinUtilsStem + StdCoinLibDir
ClpLibDir = CoinLibLocation + ClpStem + StdCoinLibDir
OsiLibDir = CoinLibLocation + OsiStem + StdCoinLibDir
CbcLibDir = CoinLibLocation + CbcStem + StdCoinLibDir
## CPLEX
#
CpxStem = '/opt/ibm/ILOG/CPLEX_Studio_Academic123/cplex/'
CpxIncDir = CpxStem + 'include/ilcplex'
CpxLibDir = CpxStem + 'lib/x86-64_sles10_4.1/static_pic'
## Gurobi
# 
GrbStem = '/opt/gurobi460/linux64/'
GrbIncDir = GrbStem + 'include'
GrbLibDir = GrbStem + 'lib'
OsiLibFlags = ['Osi', 'CoinUtils']
ClpLibFlags = ['Clp', 'OsiClp']
CbcLibFlags = ['Cbc', 'Cgl']
OsiCpxLibFlags = ['OsiCpx']
OsiGrbLibFlags = ['OsiGrb']
CpxLibFlags = ['cplex', 'ilocplex', 'pthread', 'm']
GrbLibFlags = ['gurobi_c++', 'gurobi46', 'pthread', 'm']
milpIncDirs = [CoinUtilsIncDir,
 ClpIncDir,
 OsiIncDir,
 CbcIncDir,
 CpxIncDir,
 GrbIncDir,
 GTestIncDir,
 ]
milpLibDirs = [CoinUtilsLibDir,
 ClpLibDir,
 OsiLibDir,
 CbcLibDir,
 CpxLibDir,
 GrbLibDir,
 GTestLibDir,
 ]
milpLibFlags = [OsiCpxLibFlags,
 OsiGrbLibFlags,
 CbcLibFlags,
 ClpLibFlags,
 OsiLibFlags,
 CpxLibFlags,
 GrbLibFlags,
 GTestLibFlags,
 ]
##milpSolver = env.Object('milpSolver.cpp',
 ## CPPPATH = milpIncDirs,
## LIBPATH = milpLibDirs,
## CXXFLAGS = flags)
milpSolverTest = env.Program('milpSolverUnitTest',
 ['milpSolverTest.cpp',
 'milpSolver.cpp'],
 CPPPATH = milpIncDirs,
 LIBPATH = milpLibDirs,
 LIBS = milpLibFlags,
 CXXFLAGS = flags,
 LINKFLAGS = ['-Wl,-rpath,' + OsiLibDir])
env.Depends(milpSolverTest, [GTestAllLib, GTestMainLib])
## Chemkin source directories and files
#
ChemkinSourceDir = '/mnt/hgfs/DataFromOldLaptop/Data/ModelReductionResearch/Papers/AdaptiveChemistryPaper/AdaptiveChemistry/NonOpenSource/ChemkinII/';
ChemkinSourceList = ['cklib.f', 'pcmach.f','tranlib.f']
ChemkinSourceList = [ChemkinSourceDir + FileName
 for FileName in ChemkinSourceList]
env.Depends('cklib.f','ckstrt.f')
## Cantera include directorie
#
CanteraStem = '/usr/local/cantera'
if debug:
 CanteraStem += debugString
CanteraIncDir = CanteraStem + '/include/cantera'
CanteraLibDir = CanteraStem + '/lib'
CanteraTestingFlags = ['kinetics', 'thermo', 'tpx', 'ctbase', 'm',]
CanteraLibFlags = ['user', 'oneD', 'zeroD', 'equil', 'kinetics', 'transport',
 'thermo', 'ctnumerics', 'ctmath', 'tpx', 'ctspectra',
 'converters', 'ctbase', 'cvode', 'ctlapack', 'ctblas',
 'ctf2c', 'ctcxx', 'ctf2c', 'm', 'm', 'stdc++']
CxxFortranFlags = ['g2c', 'gfortran']; 
chemSolverIncDir = [CanteraIncDir,
 StdIncDir,
 '/usr/local/include',
 GTestIncDir,
 ]
chemSolverLibDir = [StdLibDir,
 CanteraLibDir,
 GTestLibDir,
 ]
chemSolverLibFlags = [GTestLibFlags,
 CxxFortranFlags,
 CanteraLibFlags,
 ArmadilloLibFlags,
 ]
chemSolverTest = env.Program('chemSolverUnitTest',
 ['chemSolverTest.cpp',
 'chemSolver.cpp',
 'ckwrapper.f90'] + ChemkinSourceList,
 CPPPATH = chemSolverIncDir,
 LIBPATH = chemSolverLibDir,
 LIBS = chemSolverLibFlags,
 CXXFLAGS = flags,
 FORTRANFLAGS = flags,
 F90FLAGS = flags)
env.Depends(chemSolverTest, [GTestAllLib, GTestMainLib])
#env.AddPostAction(milpSolverTest, milpSolverTest[0].abspath)
testAlias = env.Alias('test', [milpSolverTest, chemSolverTest])
AlwaysBuild(testAlias)
ckInterp = env.Program('ckinterp', ChemkinSourceDir + 'ckinterp.f')
canteraGTestLibFlags = CanteraTestingFlags + GTestLibFlags
#canteraGTestLibFlags = ['kinetics', 'thermo', 'tpx',
# 'ctbase', 'm', 'gtest', 'gtest_main', 'pthread']
canteraGTest = env.Program('canteraGTest',
 'canteraGTest.cpp',
 CPPPATH = chemSolverIncDir,
 LIBPATH = chemSolverLibDir,
 LIBS = canteraGTestLibFlags,
 CXXFLAGS = flags)
env.Depends(canteraGTest, [GTestAllLib, GTestMainLib])
canteraMemTestLibFlags = CanteraTestingFlags
canteraMemTest = env.Program('canteraMemTest',
 'canteraMemTest.cpp',
 CPPPATH = chemSolverIncDir,
 LIBPATH = chemSolverLibDir,
 LIBS = canteraMemTestLibFlags,
 CXXFLAGS = flags)

Answer 1

Refactoring strategy

In addition to the comments by dietbuddha and ThorbjørnRavnAnderson, another way to refactor build scripts is to separate them into multiple files. How you do this depends on the build system.

For Make, it's as simple as using the include command, as recommended in "Recursive Make Considered Harmful". This directive works just like #include in the C preprocessor and processes the included file as if it had been cut and pasted in place of the include command. Using the include command, it's possible to refactor your main Makefile by moving modular pieces into sub-Makefiles.

CMake has a similar command.

SCons requires a similar type of approach with different commands. The basic idea of splitting up the build script into a master script and several smaller sub-scripts stays the same, but instead of directly including the text of the smaller scripts in the master build script, SCons treats the smaller scripts like separate namespaces (because SCons uses Python instead of the shell). SCons Environment objects have a method called SConscript() that enables you to import objects from an SConstruct file to subsidiary files called SConscript files that can be used to refactor your build script. The basic idea of SConscript files and the SConscript() command can be found here on the SCons Wiki. Examples of how to use SConscripts in hierarchical builds can be found here in the SCons User Guide.

Extrapolating from these three examples, it seems like the general strategy is to refactor a build script by splitting it into a master script that calls several files. How one does this is idiomatic to the particular build automation software used.

SCons Example, Revisited

Taking the SConstruct file above, I moved all of the configuration information into a module called build_config.py. All literals live in the global namespace, which could be dangerous for Python, but that's easily (though somewhat tediously) fixable. I checked ahead to make sure that I had no name clashes with __builtin__ (the Python built-in namespace, so I'm not overwriting any important objects).

## \file build_config.py
# \brief Sets configuration of file locations manually.
#
## Flags for compilers
#
flags = []
## Compile using debug versions?
#
debug = True
debugString = '-debug'
debugFlags = ['-ggdb']
dynamicLinkFlag = '-Wl,-rpath,'
if debug:
 flags += debugFlags
## Configuration information for GTest
#
GTestVersion = '1.6.0'
GTestStem = 'gtest-' + GTestVersion
GTestBuildIncDir = [GTestStem,
 GTestStem + '/include',
 ]
GTestIncDir = GTestStem + '/include/gtest'
GTestLibDir = 'lib'
GTestLibFlags = ['gtest', 'gtest_main', 'pthread']
## Configuration information for Armadillo matrix library
#
ArmadilloLibFlags = ['armadillo'];
## Locations of libraries installed on system in standard locations
#
StdIncDir = '/usr/include'
StdLibDir = '/usr/lib'
## Configuration information for COIN libraries
#
CoinUtilsVersion = '2.6.4'
ClpVersion = '1.12.0'
OsiVersion = '0.103.0'
CbcVersion = '2.5.0'
## Standard directory locations of COIN libraries, with slashes added for
# for convenience.
#
CoinLibLocation = '/usr/local/COIN/'
StdCoinIncDir = '/include/coin'
StdCoinLibDir = '/lib'
CoinUtilsStem = 'CoinUtils-' + CoinUtilsVersion
ClpStem = 'Clp-' + ClpVersion
OsiStem = 'Osi-' + OsiVersion
CbcStem = 'Cbc-' + CbcVersion
if debug:
 CoinUtilsStem += debugString
 CbcStem += debugString
 ClpStem += debugString
 OsiStem += debugString
## Build up include directory names for COIN projects from constituent parts.
#
CoinUtilsIncDir = CoinLibLocation + CoinUtilsStem + StdCoinIncDir
ClpIncDir = CoinLibLocation + ClpStem + StdCoinIncDir
OsiIncDir = CoinLibLocation + OsiStem + StdCoinIncDir
CbcIncDir = CoinLibLocation + CbcStem + StdCoinIncDir
## Build up library names from COIN projects from constituent parts
#
CoinUtilsLibDir = CoinLibLocation + CoinUtilsStem + StdCoinLibDir
ClpLibDir = CoinLibLocation + ClpStem + StdCoinLibDir
OsiLibDir = CoinLibLocation + OsiStem + StdCoinLibDir
CbcLibDir = CoinLibLocation + CbcStem + StdCoinLibDir
## CPLEX
#
CpxStem = '/opt/ibm/ILOG/CPLEX_Studio_Academic123/cplex/'
CpxIncDir = CpxStem + 'include/ilcplex'
CpxLibDir = CpxStem + 'lib/x86-64_sles10_4.1/static_pic'
## Gurobi
# 
GrbStem = '/opt/gurobi460/linux64/'
GrbIncDir = GrbStem + 'include'
GrbLibDir = GrbStem + 'lib'
OsiLibFlags = ['Osi', 'CoinUtils']
ClpLibFlags = ['Clp', 'OsiClp']
CbcLibFlags = ['Cbc', 'Cgl']
OsiCpxLibFlags = ['OsiCpx']
OsiGrbLibFlags = ['OsiGrb']
CpxLibFlags = ['cplex', 'ilocplex', 'pthread', 'm']
GrbLibFlags = ['gurobi_c++', 'gurobi46', 'pthread', 'm']
milpIncDirs = [CoinUtilsIncDir,
 ClpIncDir,
 OsiIncDir,
 CbcIncDir,
 CpxIncDir,
 GrbIncDir,
 GTestIncDir,
 ]
milpLibDirs = [CoinUtilsLibDir,
 ClpLibDir,
 OsiLibDir,
 CbcLibDir,
 CpxLibDir,
 GrbLibDir,
 GTestLibDir,
 ]
milpLibFlags = [OsiCpxLibFlags,
 OsiGrbLibFlags,
 CbcLibFlags,
 ClpLibFlags,
 OsiLibFlags,
 CpxLibFlags,
 GrbLibFlags,
 GTestLibFlags,
 ]
## Configuration information for Chemkin source directories and files
#
ChemkinSourceDir = '/mnt/hgfs/DataFromOldLaptop/Data/ModelReductionResearch/Papers/AdaptiveChemistryPaper/AdaptiveChemistry/NonOpenSource/ChemkinII/';
ChemkinSourceList = ['cklib.f', 'pcmach.f','tranlib.f']
ChemkinSourceList = [ChemkinSourceDir + FileName
 for FileName in ChemkinSourceList]
## Configuration information for Cantera
#
CanteraStem = '/usr/local/cantera'
if debug:
 CanteraStem += debugString
CanteraIncDir = CanteraStem + '/include/cantera'
CanteraLibDir = CanteraStem + '/lib'
CanteraTestingFlags = ['kinetics', 'thermo', 'tpx', 'ctbase', 'm',]
CanteraLibFlags = ['user', 'oneD', 'zeroD', 'equil', 'kinetics', 'transport',
 'thermo', 'ctnumerics', 'ctmath', 'tpx', 'ctspectra',
 'converters', 'ctbase', 'cvode', 'ctlapack', 'ctblas',
 'ctf2c', 'ctcxx', 'ctf2c', 'm', 'm', 'stdc++']
CxxFortranFlags = ['g2c', 'gfortran']; 
chemSolverIncDir = [CanteraIncDir,
 StdIncDir,
 '/usr/local/include',
 GTestIncDir,
 ]
chemSolverLibDir = [StdLibDir,
 CanteraLibDir,
 GTestLibDir,
 ]
chemSolverLibFlags = [GTestLibFlags,
 CxxFortranFlags,
 CanteraLibFlags,
 ArmadilloLibFlags,
 ]
canteraGTestLibFlags = CanteraTestingFlags + GTestLibFlags
#canteraGTestLibFlags = ['kinetics', 'thermo', 'tpx',
# 'ctbase', 'm', 'gtest', 'gtest_main', 'pthread']

The main SConstruct file calls a bunch of SConscript files to build modules containing the major features I have in my code. Moving most of the build commands to SConscript files makes the SConstruct file really simple:

## \file SConstruct
# \brief Compiles the library and compiles tests.
#
import SCons
from build_config import *
## \brief Build up directory names of each COIN library from package names
# and versions.
#
## Overall SCons environment
#
env = Environment();
## Compile Google Test from source using SConscript file.
#
GTestAllLib, GTestMainLib = env.SConscript('gtest.scons',
 exports=['env'])
## Compile MILP solver module and tests from source using SConscript file.
#
milpSolverTest = env.SConscript('milpSolver.scons',
 exports=['env'])
## Compile chemistry solver module and associated tests from source
# using SConscript file.
chemSolverTest, canteraGTest = env.SConscript('chemSolver.scons',
 exports=['env'])
## Since all tests use GTest, make the dependency of the module
# tests on the GTest libraries explicit.
env.Depends(milpSolverTest, [GTestAllLib, GTestMainLib])
env.Depends(chemSolverTest, [GTestAllLib, GTestMainLib])
env.Depends(canteraGTest, [GTestAllLib, GTestMainLib])
#env.AddPostAction(milpSolverTest, milpSolverTest[0].abspath)
testAlias = env.Alias('test', [milpSolverTest, chemSolverTest])
AlwaysBuild(testAlias)

Then all of three SConscript files have the extension .scons and break up the project into modules representing different functionality.

Google Test SConscript file:

## \file gtest.scons
# \brief SConscript file that contains information for SCons build of
# GTest. Use Python syntax highlighting for source.
from build_config import *
Import('env')
## Compile Google Test from scratch.
#
GTestAllLib = env.Library('lib/libgtest.a', 'gtest-1.6.0/src/gtest-all.cc',
 CPPPATH = GTestBuildIncDir,
 CXXFLAGS = flags)
GTestMainLib = env.Library('lib/libgtest_main.a',
 'gtest-1.6.0/src/gtest_main.cc',
 CPPPATH = GTestBuildIncDir,
 CXXFLAGS = flags)
Return('GTestAllLib', 'GTestMainLib')

Mixed-integer linear programming solver SConscript file:

## \file milpSolver.scons
# \brief SConscript file that contains information for SCons build of
# mixed-integer linear programming solver module. Use Python syntax
# highlighting for source.
from build_config import *
Import('env')
## Compile MILP solver module and tests.
#
##milpSolver = env.Object('milpSolver.cpp',
 ## CPPPATH = milpIncDirs,
## LIBPATH = milpLibDirs,
## CXXFLAGS = flags)
milpSolverTest = env.Program('milpSolverUnitTest',
 ['milpSolverTest.cpp',
 'milpSolver.cpp'],
 CPPPATH = milpIncDirs,
 LIBPATH = milpLibDirs,
 LIBS = milpLibFlags,
 CXXFLAGS = flags,
 LINKFLAGS = ['-Wl,-rpath,' + OsiLibDir])
Return('milpSolverTest')

Chemistry engine SConscript file:

## \file chemSolver.scons
# \brief SConscript file that sets up SCons build of chemistry solver module.
# Use Python syntax highlighting for source.
from build_config import *
Import('env')
## Compile CHEMKIN interpreter.
#
ckInterp = env.Program('ckinterp', ChemkinSourceDir + 'ckinterp.f')
## Enforce explicit dependence of CHEMKIN library on CHEMKIN
# parameter file 'ckstrt.f' because SCons' scanner won't pick it up.
env.Depends('cklib.f','ckstrt.f')
chemSolverTest = env.Program('chemSolverUnitTest',
 ['chemSolverTest.cpp',
 'chemSolver.cpp',
 'ckwrapper.f90'] + ChemkinSourceList,
 CPPPATH = chemSolverIncDir,
 LIBPATH = chemSolverLibDir,
 LIBS = chemSolverLibFlags,
 CXXFLAGS = flags,
 FORTRANFLAGS = flags,
 F90FLAGS = flags)
canteraGTest = env.Program('canteraGTest',
 'canteraGTest.cpp',
 CPPPATH = chemSolverIncDir,
 LIBPATH = chemSolverLibDir,
 LIBS = canteraGTestLibFlags,
 CXXFLAGS = flags)
canteraMemTestLibFlags = CanteraTestingFlags
canteraMemTest = env.Program('canteraMemTest',
 'canteraMemTest.cpp',
 CPPPATH = chemSolverIncDir,
 LIBPATH = chemSolverLibDir,
 LIBS = canteraMemTestLibFlags,
 CXXFLAGS = flags)
Return('chemSolverTest', 'canteraGTest')

The combination of these five files is probably a bit longer than the original long file, but it's easier for me to manage because I can separate the build into a configuration file, and a bunch of mostly uncoupled units, so I don't have to keep track of the whole build in my mind at one time.

Answer 2

They are functions, they just follow slightly different rules. The "functions" are often targets and rules for building. Another way to think about out it is the nodes and lines in the build DAG. The "targets" are the nodes or build artifacts and the lines are the rules on how to transform the previous nodes.

I approach refactoring build scripts first by figuring out what the DAG is. Then isolating the common rules and de-duplicate.

Here is a simplistic example:

all:
 mkdir bigfiles
 cat file1 file2 > bigfiles/bigfile1
 cat file3 file4 > bigfiles/bigfile2

What the DAG should be:

file1 \
 =-> bigfile1 \
file2 / \
 =-> all
file3 \ /
 =-> bigfile2 /
file4 /

New Rules:

 bigfiles:
 mkdir bigfiles
 bigfiles/bigfile1: bigfiles
 cat file1 file2 > bigfiles/bigfile1
 bigfiles/bigfiles2: bigfiles
 cat file3 file4 > bigfiles/bigfile2

De-duplicated:

 BIGFILES:=bigfiles/bigfile1 bigfiles/bigfile2
 bigfiles/bigfile1:=file1 file2
 bigfiles/bigfile2:=file3 file4
 .PHONY: all
 all: $(BIGFILES)
 bigfiles:
 mkdir bigfiles
 $(BIGFILES): bigfiles
 cat $($@) > $@

At the end of this exercise I currently have slightly more code than I started with. HOWEVER, You also have a more generic "function". A "function" that has been properly parametrized and is thus more maintainable and extensible as a result.

• Geoff Oxberry: +1 for the idea of looking at the DAG. I agree that there are potential opportunities to refactor there. Visualizing the DAG can be a pain in the ass. For Makefiles, there's Makefile::GraphViz, and for SCons, there's a Python script here that may require a bit of hacking, depending on the project. The graphs that script gave me were sometimes heinously large, so I found that I had to filter carefully what I wanted to visualize.
• iteratingself: You don't need to look at the entire DAG. You can just go through the Makefile of SConstruct file and target the worst parts. Identify what they are doing rework the implicit DAG and break it out. Move general rules into separate files. Organize according to data/functions/target with high cohesion like you would with a regular programming language.
• Geoff Oxberry: Yeah, that's what I was trying to do when I filtered out a lot of files, because the SConstruct dependency parser picks up a huge number of files. So I filtered those out, and used the Python script at the link to make a dotfile so I could visualize the parts of the DAG that I thought were important. It worked, and I got a small enough graph so I could pick out what was important, but I didn't see a whole lot I could use there. I suspect this strategy is a lot easier to use with Make because it's easier to find repeated rules in Makefiles, as you can see from the source I posted.

Answer 3

GNU autoconf was made to make makefiles maintainable and portable, by moving logic into a separate script.

I would consider using that the first step you could take to manage your Makefiles.

• Geoff Oxberry: I agree that GNU Autoconf was made for portability and maintainability of builds. At one point, I investigated learning Autoconf and found that it was going to take more time than I was willing to invest. (I'm finishing my PhD, so building this software needs to happen sooner rather than later.) Perhaps in the future? Portability isn't even the main issue for me, it's the length and lack of modularity in the build scripts, and I'm not sure that porting to Autoconf will solve those problems.
• user1249: Then consider showing some of your actual problems and why your current solution isn't good enough. Also, it is actually important that this is in academia.
• Geoff Oxberry: I've added examples to illustrate what I'm working with right now.