Running multiple optimization runs in parallel #756

Yes, the specific model may be multi-modal

Continuing this thread (probably not completely relevant to cmdstanpy), on the cluster when I run a single optimization process the progress is automatically synced to the standard error file when having the default show_console=False. When I now do multiple optimizations using multiprocessing, this behavior is not maintained, show_console=False only syncs the the messages "cmdstan info: process started" or something along these lines. Is there a way in which I can keep this behavior? I have used show_console=True but this syncs the output to the standard output file, which is not desired as I print some other relevant information to the standard output file.

Also would it be advisable to set STAN_NUM_THREADS to the number of threads per optimization chain, or the total number of cores available? Following the thread I posted on the stan forum, I would think it would be better to specify this as the total number of cores, creating a pool with the total number of cores, which will redistribute cores when some chains finish before others?

Can you elaborate a bit more on your question about the progress output? I'm not sure I follow.

I believe the advice is when the different runs are in different processes, like for optimization, you should set it to be the per-chain number, but I'm not entirely sure. TBB may be able to balance across multiple processes. You can always try both and monitor the number of threads in something like htop to see if you're oversubscribed

I shall try this thanks Brian.

When I only do a single optimization run without specifying the show_console arg, hence defaulting to false, the progress from stan automatically syncs to the standard error file, with some lines 'cmdstan: Info - Process started' (or something along these line) being synced to the output file (along with Process done once optimization has terminated). I can then monitor the progress whilst the chains are running through a combination of ssh and cat commands to view the standard error file, *.ER.

When I now use multiprocessing (same args), the lines 'cmdstan: Info - Process started' is synced to the standard error file instead of the output file. The progress from stan is thus not synced to any file. When I now specify show_console=True the progress from stan syncs to the standard output file, *.OU. I would prefer that the stan progress be sync to the *.ER file instead of the *.OU file when using multiprocessing

Hm, I'm not sure -- I believe cmdstanpy is capturing both standard error and standard out internally, so the fact that either are appearing in the .ER or .OU files from your job scheduler is a bit surprising to me.

I think it has to do with the multiprocessing creating several child processes? So the .OU and .ER files for the parent and child processes are not the same? Not completely sure about this however