How to run a fiber without waiting for its completion? #35

ZelphirKaltstahl commented 2020年03月01日 16:51:31 +01:00 (Migrated from github.com)

Is there any way to run a fiber (it running its thunk), without resorting to something like call-with-new-thread? Perhaps something internal to the library?

What I want to achieve is to run a fiber without the overhead of creating a new thread, trusting, that the fiber will report a result itself later on, for example using an initially given channel. Having to use call-with-new-thread or similar defeats the purpose of lightweight fibers, when the number of fibers is not known at compile time and fibers are dynamically spawned, depending on data, that the program gets as input.

I want to be able to say: "OK, new work to do came in, just start a fiber and let it report the result of the work later, when it finished. Lets go on with other stuff we need to take care of." However, run-fibers returns, when the fibers in its thunk finish, so that is not a way to go. Also using multiple schedulers does not work, as they would not know about each other and their #:parallelism limits are independent, resulting in more parallelism than wanted.

I guess you could say, that I am looking for asynchronously starting fibers and getting their results.

Is there any way to run a fiber (it running its thunk), without resorting to something like `call-with-new-thread`? Perhaps something internal to the library? What I want to achieve is to run a fiber without the overhead of creating a new thread, trusting, that the fiber will report a result itself later on, for example using an initially given channel. Having to use `call-with-new-thread` or similar defeats the purpose of lightweight fibers, when the number of fibers is not known at compile time and fibers are dynamically spawned, depending on data, that the program gets as input. I want to be able to say: "OK, new work to do came in, just start a fiber and let it report the result of the work later, when it finished. Lets go on with other stuff we need to take care of." However, `run-fibers` returns, when the fibers in its thunk finish, so that is not a way to go. Also using multiple schedulers does not work, as they would not know about each other and their `#:parallelism` limits are independent, resulting in more parallelism than wanted. I guess you could say, that I am looking for asynchronously starting fibers and getting their results.

weinholt commented 2020年03月01日 17:09:36 +01:00 (Migrated from github.com)

I might be missing something obvious, but is there a reason you don't run the whole program inside run-fibers? If you do then you can use spawn-fiber every time there is new work and then use channels as usual to get the results.

I might be missing something obvious, but is there a reason you don't run the whole program inside `run-fibers`? If you do then you can use `spawn-fiber` every time there is new work and then use channels as usual to get the results.

ZelphirKaltstahl commented 2020年03月01日 19:46:24 +01:00 (Migrated from github.com)

@weinholt That is an interesting idea, to run the whole program in run-fibers.

I think the reason why I had not considered this is, that I want to abstract from the specific parallelism implementation (parallel forms, fibers, futures, other Scheme dialect's facilities, threads, ...), that the program uses, so that one can use the program in other Scheme dialects and change just a little part, the parallelism part, which ideally would be isolated in one procedure, which is called from the program.
I guess putting a run-fibers around the main program would also not be a lot to edit when porting to another Scheme dialect, but it would definitely require touching the main program, instead of something isolated to a single module or procedure.

@weinholt That is an interesting idea, to run the whole program in `run-fibers`. I think the reason why I had not considered this is, that I want to abstract from the specific parallelism implementation (parallel forms, fibers, futures, other Scheme dialect's facilities, threads, ...), that the program uses, so that one can use the program in other Scheme dialects and change just a little part, the parallelism part, which ideally would be isolated in one procedure, which is called from the program. I guess putting a `run-fibers` around the main program would also not be a lot to edit when porting to another Scheme dialect, but it would definitely require touching the main program, instead of something isolated to a single module or procedure.