Currently all output is kept in memory with the caveat that when return = FALSE scenarios aren't returned so R's garbage collection should clear up after each scenario runs in it's usual way.
There are two options that could help:
- The first would be to implement our own cleanup whilst a scenario is running. Because we generate a DAG of the pipeline we know when objects are no longer needed. When
return = FALSEwe could remove these as appropriate during the running of a scenario. - Secondly we could always serialise and deserialise to disc for each stage of a scenario. This would be a lot less efficient but is the most extreme cleanup we could do.
Currently, I feel inclined to implement the first.
Currently all output is kept in memory with the caveat that when `return = FALSE` scenarios aren't returned so R's garbage collection should clear up after each scenario runs in it's usual way.
There are two options that could help:
- The first would be to implement our own cleanup whilst a scenario is running. Because we generate a DAG of the pipeline we know when objects are no longer needed. When `return = FALSE` we could remove these as appropriate during the running of a scenario.
- Secondly we could always serialise and deserialise to disc for each stage of a scenario. This would be a lot less efficient but is the most extreme cleanup we could do.
Currently, I feel inclined to implement the first.