-
-
Notifications
You must be signed in to change notification settings - Fork 81
question on interaction of compiler options #519
Summary:
I'm curious how the build of Cmdstan with a make/local file and the compiling of a model with supplied compiler args interact with one another
Description:
I'm constantly doing a lot of profiling on models where I'm switching at the CXXFlags/ STAN options when building Cmdstan. Since one can specify those flags in the make/local as well as when building a model via cmdstanpy, I was curious how these things interact. E.g., If I build Cmdstan:
make build
and my make/local has
STAN_OPENCL=True
Does that opencl option when building the model model override what is in make/local? So, if I set to false in cmdstanpy, it will be overridden, but if I do not set it, then it will always be tru?
Additional Information:
Provide any additional information here.
Current Version:
1.0.0
All reactions
What STAN_OPENCL does if the model is not compiled with use-opencl in stanc.
Nothing really as none of the OpenCL-related stuff will be used, so this just adds a bit more code in C++ so a bit longer compile times. You need the use-opencl flag in stanc3 in order to generate the model with the OpenCL-related constructs (matrix_cl class, etc) and setting STAN_OPENCL then enables the OpenCL code in Stan Math so that the generated C++ actually compiles.
To sum up:
- no use-opencl stanc3 flag & STAN_OPENCL=False -> normal sequential case
- no use-opencl stanc3 flag & STAN_OPENCL=True -> same as sequential code, a bit longer compile times excepted
- use-opencl stanc3 flag & STAN_OPENCL= False -> if...
Replies: 7 comments
Good question. ( I don't have an answer to this)
which one is preferred a cmdline argument or the make/local file.
(Question for anyone needing to debug this, is there a way to print in somewhere while building is going on)
All reactions
I have noticed that if I turn on OpenCL in the make/local, but say nothing in the cmdline, I get the following output during the build:
WARNING:cmdstanpy:g++ -funroll-loops -mfpmath=sse -DEIGEN_USE_BLAS -DEIGEN_USE_LAPACKE -mtune=native -march=native -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes -I stan/lib/stan_math/lib/opencl_3.0.0 -DSTAN_THREADS -I stan/lib/stan_math/lib/tbb_2020.3/include -flto -fuse-linker-plugin -fdevirtualize-at-ltrans -fweb -fivopts -ftree-loop-linear -floop-strip-mine -floop-block -floop-nest-optimize -ftree-vectorize -ftree-loop-distribution -funroll-loops -floop-unroll-and-jam -fsplit-loops -fvisibility=hidden -fvisibility-inlines-hidden -DSTAN_NO_RANGE_CHECKS -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials -DBOOST_DISABLE_ASSERTS -DSTAN_OPENCL -DOPENCL_DEVICE_ID=0 -DOPENCL_PLATFORM_ID=0 -DCL_HPP_TARGET_OPENCL_VERSION=120 -DCL_HPP_MINIMUM_OPENCL_VERSION=120 -DCL_HPP_ENABLE_EXCEPTIONS -Wno-ignored-attributes -DSTAN_NO_RANGE_CHECKS -c -MT stan/src/stan/model/model_header_threads_opencl_nochecks.hpp.gch -MT stan/src/stan/model/model_header_threads_opencl_nochecks.d -MM -E -MG -MP -MF stan/src/stan/model/model_header_threads_opencl_nochecks.d stan/src/stan/model/model_header.hpp --- Translating Stan model to C++ code --- bin/stanc --include_paths=/home/jburgess/sw/whimstan/whimstan/stan_code --o=/home/jburgess/sw/whimstan/whimstan/stan_code/whim.hpp /home/jburgess/sw/whimstan/whimstan/stan_code/whim.stan --include_paths is deprecated and Will be removed in Stan 2.32.0. Please use --include-paths. Warning in '/home/jburgess/sw/whimstan/whimstan/stan_code/whim.stan', line 43, column 2: Declaration of arrays by placing brackets after a variable name is deprecated and will be removed in Stan 2.32.0. Instead use the array keyword before the type. This can be changed automatically using the auto-format flag to stanc --- Compiling, linking C++ code --- g++ -funroll-loops -mfpmath=sse -DEIGEN_USE_BLAS -DEIGEN_USE_LAPACKE -mtune=native -march=native -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes -I stan/lib/stan_math/lib/opencl_3.0.0 -DSTAN_THREADS -I stan/lib/stan_math/lib/tbb_2020.3/include -flto -fuse-linker-plugin -fdevirtualize-at-ltrans -fweb -fivopts -ftree-loop-linear -floop-strip-mine -floop-block -floop-nest-optimize -ftree-vectorize -ftree-loop-distribution -funroll-loops -floop-unroll-and-jam -fsplit-loops -fvisibility=hidden -fvisibility-inlines-hidden -DSTAN_NO_RANGE_CHECKS -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials -DBOOST_DISABLE_ASSERTS -DSTAN_OPENCL -DOPENCL_DEVICE_ID=0 -DOPENCL_PLATFORM_ID=0 -DCL_HPP_TARGET_OPENCL_VERSION=120 -DCL_HPP_MINIMUM_OPENCL_VERSION=120 -DCL_HPP_ENABLE_EXCEPTIONS -Wno-ignored-attributes -DSTAN_NO_RANGE_CHECKS -c -Wno-ignored-attributes -fweb -fivopts -ftree-loop-linear -floop-strip-mine -floop-block -floop-nest-optimize -ftree-vectorize -ftree-loop-distribution -funroll-loops -floop-unroll-and-jam -fsplit-loops -fvisibility=hidden -fvisibility-inlines-hidden -flto -fuse-linker-plugin -fdevirtualize-at-ltrans -x c++ -o /home/jburgess/sw/whimstan/whimstan/stan_code/whim.o /home/jburgess/sw/whimstan/whimstan/stan_code/whim.hpp g++ -funroll-loops -mfpmath=sse -DEIGEN_USE_BLAS -DEIGEN_USE_LAPACKE -mtune=native -march=native -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes -I stan/lib/stan_math/lib/opencl_3.0.0 -DSTAN_THREADS -I stan/lib/stan_math/lib/tbb_2020.3/include -flto -fuse-linker-plugin -fdevirtualize-at-ltrans -fweb -fivopts -ftree-loop-linear -floop-strip-mine -floop-block -floop-nest-optimize -ftree-vectorize -ftree-loop-distribution -funroll-loops -floop-unroll-and-jam -fsplit-loops -fvisibility=hidden -fvisibility-inlines-hidden -DSTAN_NO_RANGE_CHECKS -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials -DBOOST_DISABLE_ASSERTS -DSTAN_OPENCL -DOPENCL_DEVICE_ID=0 -DOPENCL_PLATFORM_ID=0 -DCL_HPP_TARGET_OPENCL_VERSION=120 -DCL_HPP_MINIMUM_OPENCL_VERSION=120 -DCL_HPP_ENABLE_EXCEPTIONS -Wno-ignored-attributes -DSTAN_NO_RANGE_CHECKS -Wl,-L,"/home/jburgess/.cmdstan/cmdstan/stan/lib/stan_math/lib/tbb" -Wl,-rpath,"/home/jburgess/.cmdstan/cmdstan/stan/lib/stan_math/lib/tbb" -flto -fuse-linker-plugin -fdevirtualize-at-ltrans /home/jburgess/sw/whimstan/whimstan/stan_code/whim.o src/cmdstan/main_threads_opencl_nochecks.o -lblas -llapack -llapacke -lOpenCL -Wl,-L,"/home/jburgess/.cmdstan/cmdstan/stan/lib/stan_math/lib/tbb" -Wl,-rpath,"/home/jburgess/.cmdstan/cmdstan/stan/lib/stan_math/lib/tbb" stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_nvecserial.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_cvodes.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_idas.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_kinsol.a stan/lib/stan_math/lib/tbb/libtbb.so.2 -o /home/jburgess/sw/whimstan/whimstan/stan_code/whim rm -f /home/jburgess/sw/whimstan/whimstan/stan_code/whim.o
I do see the OpenCL stuff popping up.
All reactions
CmdStanPy does do something special related to opencl, which is if you pass the use-opencl flag to stanc, it will also turn on STAN_OPENCL in the C++ options.
There are two things I'm not sure about -
- How command line arguments and make/local compose
- What STAN_OPENCL does if the model is not compiled with
use-openclin stanc.
@rok-cesnovar - do you have a better answer for either of those?
All reactions
command line arguments to make override make/local argument.
https://ftp.gnu.org/old-gnu/Manuals/make-3.79.1/html_chapter/make_9.html#SEC90
here's what's in the CmdStan docs w/r/t OpenCL - @rok-cesnovar are these correct? https://mc-stan.org/docs/cmdstan-guide/parallelization.html#opencl
All reactions
What STAN_OPENCL does if the model is not compiled with use-opencl in stanc.
Nothing really as none of the OpenCL-related stuff will be used, so this just adds a bit more code in C++ so a bit longer compile times. You need the use-opencl flag in stanc3 in order to generate the model with the OpenCL-related constructs (matrix_cl class, etc) and setting STAN_OPENCL then enables the OpenCL code in Stan Math so that the generated C++ actually compiles.
To sum up:
- no use-opencl stanc3 flag & STAN_OPENCL=False -> normal sequential case
- no use-opencl stanc3 flag & STAN_OPENCL=True -> same as sequential code, a bit longer compile times excepted
- use-opencl stanc3 flag & STAN_OPENCL= False -> if any OpenCL code was generated in stanc3, the C++ will not compile
- use-opencl stanc3 flag & STAN_OPENCL= True -> OpenCL model
are these correct?
They are.
@grburgess if you are ever in doubt about what gets passed to the compiler, I suggest using ̇make print-variable` command. Its precisely used for debugging these kind of stuff.
I set STAN_OPENCL=false in make/local and run:
rok@rok-Z97X-UD5H:~/Desktop/cmdstan$ make print-STAN_OPENCL
STAN_OPENCL = false
rok@rok-Z97X-UD5H:~/Desktop/cmdstan$ make STAN_OPENCL=true print-STAN_OPENCL
STAN_OPENCL = true
All reactions
-
👍 2
And make is its own beast, I never assume I know anything without triple-checking and even then I don't trust it haha
All reactions
-
👍 2 -
😄 1
Awesome. I think I understand the workflow now. Thanks!