"kevent: Bad file descriptor" after fork on freebsd #150

Dariqq commented 2025年10月06日 15:19:50 +02:00

Hi,

I encountered this while looking into the daemonize action of shepherd.

On freebsd if a program calls fork after a run-fibers results in lots of

[warn] kevent: Bad file descriptor
[warn] kevent: Bad file descriptor
[warn] kevent: Bad file descriptor
[warn] kevent: Bad file descriptor

messages and fibers/guile utilizing 100% of the cpu.

A minimal example is

(use-modules (fibers)
			 (fibers conditions))
(run-fibers
 (lambda ()
 (if (zero? (primitive-fork))
	 ;; wait forever
 (wait (make-condition))
 (primitive-exit 0)))
 #:parallelism 1
 #:hz 0)

% uname
FreeBSD
% guile3 example.scm
[warn] kevent: Bad file descriptor
[warn] kevent: Bad file descriptor
[warn] kevent: Bad file descriptor
[....]

The manual page for kevent/kqueue says

> The kqueue() system call creates a new kernel event queue and returns a descriptor. 
> The queue is not inherited by a child created with fork(2).

which might be the cause. However I am not sure if this a problem in libevent or how fibers uses libevent

FreeBSD 14.3
libevent 2.1.12 (from ports)
guile 3.0.10 (from ports, recompiled with additional ac_cv_func_posix_spawn_file_actions_addclosefrom_np=no)
fibers 4d37f56ef38f653841d151104262928b3a935396

Hi, I encountered this while looking into the `daemonize` action of shepherd. On freebsd if a program calls `fork` after a `run-fibers` results in lots of ```text [warn] kevent: Bad file descriptor [warn] kevent: Bad file descriptor [warn] kevent: Bad file descriptor [warn] kevent: Bad file descriptor ``` messages and fibers/guile utilizing 100% of the cpu. A minimal example is ```scm (use-modules (fibers) (fibers conditions)) (run-fibers (lambda () (if (zero? (primitive-fork)) ;; wait forever (wait (make-condition)) (primitive-exit 0))) #:parallelism 1 #:hz 0) ``` ```sh % uname FreeBSD % guile3 example.scm [warn] kevent: Bad file descriptor [warn] kevent: Bad file descriptor [warn] kevent: Bad file descriptor [....] ``` The manual page for kevent/kqueue says ``` > The kqueue() system call creates a new kernel event queue and returns a descriptor. > The queue is not inherited by a child created with fork(2). ``` which might be the cause. However I am not sure if this a problem in libevent or how fibers uses libevent ``` FreeBSD 14.3 libevent 2.1.12 (from ports) guile 3.0.10 (from ports, recompiled with additional ac_cv_func_posix_spawn_file_actions_addclosefrom_np=no) fibers 4d37f56ef38f653841d151104262928b3a935396 ```

civodul commented 2025年10月06日 22:50:59 +02:00

Hey @Dariqq,

It sounds like libevent should use pthread_atfork to create a fresh event queue in the child?

Maybe worth checking if a minimal example in C using libevent exhibits this problem, in which case this could be reported to libevent developers.

Thanks!

Hey @Dariqq, It sounds like libevent should use `pthread_atfork` to create a fresh event queue in the child? Maybe worth checking if a minimal example in C using libevent exhibits this problem, in which case this could be reported to libevent developers. Thanks!

Dariqq commented 2025年10月07日 09:46:02 +02:00

Hi @civodul

I read through the libevent documentation and there is an environment variable EVENT_NOKQUEUE one can set to disable the kqueue backend for libevent.

With this I get no problem with a daemonizing shepherd both in the daemonize test and in the wild.

I'll try to reproduce with libevent directly and will report to them.

Hi @civodul I read through the libevent documentation and there is an environment variable `EVENT_NOKQUEUE` one can set to disable the kqueue backend for libevent. With this I get no problem with a daemonizing shepherd both in the `daemonize` test and in the wild. I'll try to reproduce with libevent directly and will report to them.

Dariqq commented 2025年10月07日 14:12:03 +02:00

@civodul

There is already a mechanism in libvent to reinit events specifically after fork with event_reinit (see https://libevent.org/libevent-book/Ref2_eventbase.html) .

As fibers does not seem to use it currently I think this is more of a problem here than in libevent.

But I am not sure yet how to implement it.

I am not yet super familiar with the fibers architecture but iiuc there is the current-scheduler which has the scheduler-events-impl field which contains a <libevt> record.

The <libevt> record has the libevt-ls field which contains the list of a pointer to a libevt_data and the wait_data which I dont think I care about for this. The libevt-data has the the base member which is the event_base from libevent.

I think this is what I need to call the event_reinit on.

I managed to make it work by adding a events-impl-reinit! (with a noop for epoll) and then a public scheduler-reinit! that calls this on the libevt-ls.

Then in shepherd I added the scheduler-reinit! on the current-scheduler to the %post-daemonize-hook and libevent is not complaining anymore.

This is not really elegant and more of a proof of concept that this actually works

@civodul There is already a mechanism in libvent to reinit events specifically after fork with `event_reinit` (see https://libevent.org/libevent-book/Ref2_eventbase.html) . As fibers does not seem to use it currently I think this is more of a problem here than in libevent. But I am not sure yet how to implement it. I am not yet super familiar with the fibers architecture but iiuc there is the `current-scheduler` which has the `scheduler-events-impl` field which contains a `<libevt>` record. The `<libevt>` record has the `libevt-ls` field which contains the list of a pointer to a `libevt_data` and the `wait_data` which I dont think I care about for this. The `libevt-data` has the the `base` member which is the `event_base` from libevent. I think this is what I need to call the `event_reinit` on. I managed to make it work by adding a `events-impl-reinit!` (with a noop for epoll) and then a public `scheduler-reinit!` that calls this on the `libevt-ls`. Then in shepherd I added the `scheduler-reinit!` on the `current-scheduler` to the `%post-daemonize-hook` and libevent is not complaining anymore. This is not really elegant and more of a proof of concept that this actually works

civodul commented 2025年10月07日 14:33:37 +02:00

@Dariqq Could Fibers install an atfork handler to call event_reinit automatically?

It doesn’t seem right for the application to deal with that.

@Dariqq Could Fibers install an atfork handler to call `event_reinit` automatically? It doesn’t seem right for the application to deal with that.

Dariqq commented 2025年10月07日 16:55:15 +02:00

I think that works as well.

I added it in the scheduler constructor (i am hoping that the pointer to the event_base stays the the same between the constructor getting called and fork).

But now I need the entire (system foreign) module to get the pthread_atfork ffi and for procedure->pointer for the thunk around events-impl-reinit! .

Also when I tried it on linux I am unable to find the atfork symbol from within guile (#f, libc, pthread)

I think that works as well. I added it in the scheduler constructor (i am hoping that the pointer to the `event_base` stays the the same between the constructor getting called and `fork`). But now I need the entire `(system foreign)` module to get the pthread_atfork ffi and for `procedure->pointer` for the thunk around `events-impl-reinit!` . Also when I tried it on linux I am unable to find the atfork symbol from within guile (#f, libc, pthread)

Dariqq commented 2025年10月07日 18:30:04 +02:00

It also looks like the atfork is also creating new issues:

With it the shepherd/tests/services/timer now has lots of the same [kevent] EBADF when triggering the never-timer.
Also the shepherd/tests/forking-service fails with it so it might get called when we dont want it.

When only the %post-daemonize-hook calls it, this is not a problem.

It also looks like the atfork is also creating new issues: With it the `shepherd/tests/services/timer` now has lots of the same `[kevent] EBADF` when triggering the never-timer. Also the shepherd/tests/forking-service fails with it so it might get called when we dont want it. When only the %post-daemonize-hook calls it, this is not a problem.

Dariqq commented 2025年10月08日 15:06:20 +02:00

I put the thing I tried yesterday into a WIP pr to have improve the discussion on how to resolve this

I put the thing I tried yesterday into a WIP pr to have improve the discussion on how to resolve this