WIP: timer-wheel: Allow gaps to be skipped. #177

civodul commented 2026年04月06日 21:49:39 +02:00

• fibers/timer-wheel.scm ()[entry-count]: New field.
  (make-timer-wheel): Initialize it.
  (increment-entry-count!, decrement-entry-count!): New procedures.
  (timer-wheel-remove!): Call ‘decrement-entry-count!’.
  (timer-wheel-advance!): Add clause for wheels with ‘entry-count’ = 0.
  Add call to ‘decrement-entry-count!’ in the non-empty case. Adjust time
  base based on ‘outer’.
  [tick-outer!]: Add call to ‘increment-entry-count!’.
  Adjust main body to reload the time base and current pointer of ‘wheel’.
• tests/timer-wheel.scm (test-advancing-gap): New procedure.

Fixes: guile/fibers#89

* fibers/timer-wheel.scm (<timer-wheel>)[entry-count]: New field. (make-timer-wheel): Initialize it. (increment-entry-count!, decrement-entry-count!): New procedures. (timer-wheel-remove!): Call ‘decrement-entry-count!’. (timer-wheel-advance!): Add clause for wheels with ‘entry-count’ = 0. Add call to ‘decrement-entry-count!’ in the non-empty case. Adjust time base based on ‘outer’. [tick-outer!]: Add call to ‘increment-entry-count!’. Adjust main body to reload the time base and current pointer of ‘wheel’. * tests/timer-wheel.scm (test-advancing-gap): New procedure. Fixes: guile/fibers#89

civodul commented 2026年04月06日 21:53:02 +02:00

This seems to fix #89 without breaking anything else. I had a go at running the Shepherd test suite against it; I see intermittent failures but I’m not sure if I can attribute them to this change (there are a few flaky tests).

This change works by adding an entry counter to <timer-wheel>. When a wheel has zero entries, timer-wheel-advance! can skip straight to the required time or to the outer wheel.

I’d like to have more eyeballs on this, especially from @andywingo and @stund, and also more testing.

This *seems* to fix #89 without breaking anything else. I had a go at running the Shepherd test suite against it; I see intermittent failures but I’m not sure if I can attribute them to this change (there are a few flaky tests). This change works by adding an entry counter to `<timer-wheel>`. When a wheel has zero entries, `timer-wheel-advance!` can skip straight to the required time or to the outer wheel. I’d like to have more eyeballs on this, especially from @andywingo and @stund, and also more testing.

alexsh commented 2026年04月09日 08:18:09 +02:00

Hello, I am affected by the issue and I would like to help test it. How do I go about that? Where to start? I'm new to Guix.
I have: /gnu/store/f5lvaqcss9pg3al6np9r8gmiia19h755-shepherd-1.0.9/bin/shepherd which depends on guile-fibers@1.4.2.

Hello, I am affected by the issue and I would like to help test it. How do I go about that? Where to start? I'm new to Guix. I have: `/gnu/store/f5lvaqcss9pg3al6np9r8gmiia19h755-shepherd-1.0.9/bin/shepherd` which depends on `guile-fibers@1.4.2`.

civodul commented 2026年04月10日 01:19:00 +02:00

Hi @alexsh,

You’ll need to define a variant of the Shepherd along these lines:

(define guile-fibers-with-fix
 (let ((commit "0183210cad3320f4bca59e3c6ad1deb98aa01dc0")
 (revision "0"))
 (package/inherit guile-fibers-1.4
 (name "guile-fibers")
 (version (git-version "1.4.2" revision commit))
 (source
 (origin
 (method git-fetch)
 (uri (git-reference
 (url (package-home-page guile-fibers-1.4))
 (commit commit)))
 (file-name (git-file-name name version))
 (sha256
 (base32 "1ipq0vx7r7sql68x5qbrslgx3kmqaaqziwmhhh0qwpaq3bhjad3v")))))))
(define shepherd-with-fibers-fix
 (package/inherit shepherd-1.0
 (name "shepherd-under-test")
 (native-inputs
 (modify-inputs native-inputs
 (replace "guile-fibers" guile-fibers-with-fix)))))
(operating-system
 ;; ...
 (essential-services
 (modify-services (operating-system-default-essential-services
 this-operating-system)
 (shepherd-root-service-type
 config =>
 (shepherd-configuration
 (inherit config)
 (shepherd shepherd-with-fibers-fix))))))

(I have not tested this yet.)

You can probably test in guix system vm before testing on the bare metal. Or you can test similarly for Guix Home.

Thanks for offering to help!

Hi @alexsh, You’ll need to define a variant of the Shepherd along these lines: ```scheme (define guile-fibers-with-fix (let ((commit "0183210cad3320f4bca59e3c6ad1deb98aa01dc0") (revision "0")) (package/inherit guile-fibers-1.4 (name "guile-fibers") (version (git-version "1.4.2" revision commit)) (source (origin (method git-fetch) (uri (git-reference (url (package-home-page guile-fibers-1.4)) (commit commit))) (file-name (git-file-name name version)) (sha256 (base32 "1ipq0vx7r7sql68x5qbrslgx3kmqaaqziwmhhh0qwpaq3bhjad3v"))))))) (define shepherd-with-fibers-fix (package/inherit shepherd-1.0 (name "shepherd-under-test") (native-inputs (modify-inputs native-inputs (replace "guile-fibers" guile-fibers-with-fix))))) (operating-system ;; ... (essential-services (modify-services (operating-system-default-essential-services this-operating-system) (shepherd-root-service-type config => (shepherd-configuration (inherit config) (shepherd shepherd-with-fibers-fix)))))) ``` (I have not tested this yet.) You can probably test in `guix system vm` before testing on the bare metal. Or you can test similarly for Guix Home. Thanks for offering to help!

alexsh commented 2026年04月10日 09:52:08 +02:00

Thanks for the detailed instructions — that's exactly what I was missing! I will give it a try soon and let you know.

Thanks for the detailed instructions — that's exactly what I was missing! I will give it a try soon and let you know.

burban commented 2026年04月12日 00:19:42 +02:00

Hi civodul!

I have played a bit with some of your WIP to get rid of the 100% CPU issue (happening at clock change or after the system has run for a few days).

Currently I run latest git fibers+shepherd as PID 1 with the following WIP added:

fibers: WIP#166 + WIP#177
shepherd: WIP#104

and the results per architecture are:

amd64, good RTC: fine (actually never was a serious issue)

arm64, good RTC, as LXC container under OpenWRT: fine (really bad 100% CPU issue before)

armhf, bad RTC: fine (used to lock computer before at boot, as shepherd was probably trying so catch up from 1 January 1970)

So you are definiteley on the right track! I guess the monotonic clock WIP could actually be dropped.

Sincerely.

Hi civodul! I have played a bit with some of your WIP to get rid of the 100% CPU issue (happening at clock change or after the system has run for a few days). Currently I run latest git fibers+shepherd as PID 1 with the following WIP added: fibers: WIP#166 + WIP#177 shepherd: WIP#104 and the results per architecture are: amd64, good RTC: fine (actually never was a serious issue) arm64, good RTC, as LXC container under OpenWRT: fine (really bad 100% CPU issue before) armhf, bad RTC: fine (used to lock computer before at boot, as shepherd was probably trying so catch up from 1 January 1970) So you are definiteley on the right track! I guess the monotonic clock WIP could actually be dropped. Sincerely.

stund commented 2026年04月12日 04:14:18 +02:00

Glad this seems to work, but a bit dismayed at the added complexity versus using a monotonic clock. IMO the latter provides a conceptually simpler solution which also fixes shepherd/shepherd#105, and would effectively address another fundamental issue:

Fibers sleep breaks the semantics of Guile core sleep, and sleep(3), because it is affected by clock jumps.

I doubt this was intentional, but more likely an accidental by-product of Guile's schizophrenic get-internal-real-time API. Only @andywingo knows?

Glad this seems to work, but a bit dismayed at the added complexity versus using a [monotonic clock](https://codeberg.org/guile/fibers/pulls/166). IMO the latter provides a conceptually simpler solution which also fixes shepherd/shepherd#105, and would effectively address another fundamental issue: Fibers `sleep` breaks the semantics of Guile core `sleep`, and `sleep(3)`, because it is affected by clock jumps. I doubt this was intentional, but more likely an accidental by-product of Guile's schizophrenic `get-internal-real-time` API. Only @andywingo knows?

alexsh commented 2026年04月12日 10:00:34 +02:00

@civodul I'm trying to add the above config to my /etc/config.scm, but seems like I'm missing the required use-modules incantation, and I cannot seem to just figure it out on my own:

/sudo:root@localhost:/etc/ #$ guix system reconfigure /etc/config.scm
guix system: error: failed to load '/etc/config.scm':
ice-9/boot-9.scm:3384:6: In procedure resolve-interface:
no code for module (gnu packages fibers)

I've tried all variants I could think of, the latest being:

(use-modules (gnu)
 (guix packages)
 (gnu packages fibers))

@burban I see that you managed to test it already: could you please share use-modules that worked for you?

@civodul I'm trying to add the above config to my `/etc/config.scm`, but seems like I'm missing the required `use-modules` incantation, and I cannot seem to just figure it out on my own: ```sh /sudo:root@localhost:/etc/ #$ guix system reconfigure /etc/config.scm guix system: error: failed to load '/etc/config.scm': ice-9/boot-9.scm:3384:6: In procedure resolve-interface: no code for module (gnu packages fibers) ``` I've tried all variants I could think of, the latest being: ```scm (use-modules (gnu) (guix packages) (gnu packages fibers)) ``` @burban I see that you managed to test it already: could you please share `use-modules` that worked for you?

burban commented 2026年04月12日 10:28:35 +02:00

Hi!

To stund: without WIP#177 (so with only monotonic patch), the 100% CPU issue was still obviously present on arm64.

To alexsh: I am running shepherd on Debian, see my ticket shepherd/shepherd#97.
In the middle of that thread, I have my config files shepherd.tgz (link: https://codeberg.org/attachments/9e613b2b-cb8f-46f2-9a0e-a574cf8a7305) which may help you, especially top of shepherd.scm; but as Guix and Debian are not the same beast, this is probably not a direct drop in

Cheers.

Hi! To stund: without WIP#177 (so with only monotonic patch), the 100% CPU issue was still obviously present on arm64. To alexsh: I am running shepherd on Debian, see my ticket https://codeberg.org/shepherd/shepherd/issues/97. In the middle of that thread, I have my config files shepherd.tgz (link: https://codeberg.org/attachments/9e613b2b-cb8f-46f2-9a0e-a574cf8a7305) which may help you, especially top of shepherd.scm; but as Guix and Debian are not the same beast, this is probably not a direct drop in Cheers.

alexsh commented 2026年04月12日 10:42:07 +02:00

Ah, figured that one out now, thanks for the extra pointers!

(use-modules (gnu)
 (guix packages)
 (guix git-download)
 (gnu packages)
 (gnu packages admin)
 (gnu packages guile-xyz))

I was confused by the build system's suggestion to add guile-xyz, assuming that's not an actual package name, but rather placeholder %)

Ah, figured that one out now, thanks for the extra pointers! ```scm (use-modules (gnu) (guix packages) (guix git-download) (gnu packages) (gnu packages admin) (gnu packages guile-xyz)) ``` I was confused by the build system's suggestion to add `guile-xyz`, assuming that's not an actual package name, but rather placeholder %)

alexsh commented 2026年04月12日 11:07:06 +02:00

Hm, not sure if this has any relation to what I attempted to achieve, but now it fails with the following message:

root# guix system reconfigure /etc/config.scm
guix system: warning: Consider running 'guix pull' before 'reconfigure'.
guix system: warning: Failing to do that may downgrade your system!
guix system: error: aborting reconfiguration because commit 520785e315eddbe47199ac557e88e60eca3ae97c of channel 'guix' is not a descendant of 02c36a8ad87a4018d310fde26d5e0cd07fa7066b
hint: Use `--allow-downgrades' to force this downgrade.

I did try guix pull, but after that the same error is still there.

Hm, not sure if this has any relation to what I attempted to achieve, but now it fails with the following message: ```sh root# guix system reconfigure /etc/config.scm guix system: warning: Consider running 'guix pull' before 'reconfigure'. guix system: warning: Failing to do that may downgrade your system! guix system: error: aborting reconfiguration because commit 520785e315eddbe47199ac557e88e60eca3ae97c of channel 'guix' is not a descendant of 02c36a8ad87a4018d310fde26d5e0cd07fa7066b hint: Use `--allow-downgrades' to force this downgrade. ``` I did try `guix pull`, but after that the same error is still there.

alexsh commented 2026年04月12日 11:34:27 +02:00

Apparently my mistake was to run the guix pull command from the root shell. Now that this is out of the way, I'm getting some new errors, this time related to the additions to the config.scm suggested above by @civodul

$ sudo guix system reconfigure /etc/config.scm
Password: 
Backtrace:
 18 (primitive-load "/home/ash/.config/guix/current/bin/guix")
In guix/ui.scm:
 2402:7 17 (run-guix . _)
 2365:10 16 (run-guix-command _ . _)
In ice-9/boot-9.scm:
 1784:12 15 (with-exception-handler _ _ #:unwind? _ # _)
In guix/status.scm:
 862:3 14 (_)
 842:4 13 (call-with-status-report _ _)
In guix/scripts/system.scm:
 1351:4 12 (_)
In ice-9/boot-9.scm:
 1784:12 11 (with-exception-handler _ _ #:unwind? _ # _)
In guix/store.scm:
 512:37 10 (thunk)
 1127:8 9 (call-with-build-handler #<procedure 7f5397476750 at g...> ...)
 1864:25 8 (run-with-store #<store-connection 256.100 7f53974753c0> ...)
In guix/scripts/system.scm:
 735:8 7 (_ #<store-connection 256.100 7f53974753c0>)
In gnu/system.scm:
 1345:19 6 (operating-system-derivation _)
 894:11 5 (operating-system-services #<<operating-system> kernel:...>)
In /etc/config.scm:
 70:21 4 (essential-services #<<operating-system> kernel: #<pack...>)
In ice-9/boot-9.scm:
 1705:22 3 (raise-exception _ #:continuable? _)
In guix/store.scm:
 519:30 2 (_ _)
In ice-9/boot-9.scm:
 1705:22 1 (raise-exception _ #:continuable? _)
In guix/ui.scm:
 921:18 0 (_ _)
guix/ui.scm:921:18: error: shepherd-root-service-type: unbound variable

What do I do now?

Apparently my mistake was to run the `guix pull` command from the root shell. Now that this is out of the way, I'm getting some new errors, this time related to the additions to the `config.scm` suggested above by @civodul ```sh $ sudo guix system reconfigure /etc/config.scm Password: Backtrace: 18 (primitive-load "/home/ash/.config/guix/current/bin/guix") In guix/ui.scm: 2402:7 17 (run-guix . _) 2365:10 16 (run-guix-command _ . _) In ice-9/boot-9.scm: 1784:12 15 (with-exception-handler _ _ #:unwind? _ # _) In guix/status.scm: 862:3 14 (_) 842:4 13 (call-with-status-report _ _) In guix/scripts/system.scm: 1351:4 12 (_) In ice-9/boot-9.scm: 1784:12 11 (with-exception-handler _ _ #:unwind? _ # _) In guix/store.scm: 512:37 10 (thunk) 1127:8 9 (call-with-build-handler #<procedure 7f5397476750 at g...> ...) 1864:25 8 (run-with-store #<store-connection 256.100 7f53974753c0> ...) In guix/scripts/system.scm: 735:8 7 (_ #<store-connection 256.100 7f53974753c0>) In gnu/system.scm: 1345:19 6 (operating-system-derivation _) 894:11 5 (operating-system-services #<<operating-system> kernel:...>) In /etc/config.scm: 70:21 4 (essential-services #<<operating-system> kernel: #<pack...>) In ice-9/boot-9.scm: 1705:22 3 (raise-exception _ #:continuable? _) In guix/store.scm: 519:30 2 (_ _) In ice-9/boot-9.scm: 1705:22 1 (raise-exception _ #:continuable? _) In guix/ui.scm: 921:18 0 (_ _) guix/ui.scm:921:18: error: shepherd-root-service-type: unbound variable ``` What do I do now?

alexsh commented 2026年04月15日 13:03:10 +02:00

I'm kind of stuck here, so would appreciate any help. :)

I'm kind of stuck here, so would appreciate any help. :)

civodul commented 2026年04月18日 17:24:45 +02:00

Glad this seems to work, but a bit dismayed at the added complexity versus using a monotonic clock.

@stund Yeah, I think we could have both because it’s a problem that timer-wheel-advance! can spin forever.

Then yes, we would need sleep to operate on a monotonic clock, but we also need timer-operation to operate on real time (the latter is what the Shepherd would use; the "real time emulation" in shepherd/shepherd#104 shouldn’t be needed, it’s a very basic use case). I don’t know how to change the scheduler to deal with two timer wheels though...

> Glad this seems to work, but a bit dismayed at the added complexity versus using a monotonic clock. @stund Yeah, I think we could have both because it’s a problem that `timer-wheel-advance!` can spin forever. Then yes, we would need `sleep` to operate on a monotonic clock, but we also need `timer-operation` to operate on real time (the latter is what the Shepherd would use; the "real time emulation" in shepherd/shepherd#104 shouldn’t be needed, it’s a very basic use case). I don’t know how to change the scheduler to deal with two timer wheels though...

civodul commented 2026年04月18日 21:22:11 +02:00

All this got me thinking: how do people wait for a "real time" expiry? There seems to be only one good solution: timerfd, which is Linux-specific.

But then, how did good’ol cron implementations deal with that? Apparently they didn’t: anacron just uses sleep:

github.com/cronie-crond/cronie@5f9f16b566/anacron/main.c (L508-L513)

It has a simple heuristic to detect cases where the real time clock jumps forward:

github.com/cronie-crond/cronie@5f9f16b566/anacron/main.c (L391-L396)

All this got me thinking: how do people wait for a "real time" expiry? There seems to be only one good solution: timerfd, which is Linux-specific. But then, how did good’ol cron implementations deal with that? Apparently they didn’t: anacron just uses `sleep`: https://github.com/cronie-crond/cronie/blob/5f9f16b5663becefdd0dd70df31c0ef5ac36f943/anacron/main.c#L508-L513 It has a simple heuristic to detect cases where the real time clock jumps forward: https://github.com/cronie-crond/cronie/blob/5f9f16b5663becefdd0dd70df31c0ef5ac36f943/anacron/main.c#L391-L396

stund commented 2026年04月19日 16:42:26 +02:00

@civodul wrote in #177 (comment):

[...] But then, how did good’ol cron implementations deal with that? Apparently they didn’t: anacron just uses sleep:

github.com/cronie-crond/cronie@5f9f16b566/anacron/main.c (L508-L513)

It has a simple heuristic to detect cases where the real time clock jumps forward:

github.com/cronie-crond/cronie@5f9f16b566/anacron/main.c (L391-L396)

That sleep implements a job's optional "delay in minutes", per anacrontab(5). But anacron is intended for jobs of at most daily frequency on systems that are intermittently running. It's part of -- and ordinarily run by -- cron (in cronie).

The sleep loop that actually controls the scheduling in cron itself is in cron.c's main, starting here:

github.com/cronie-crond/cronie@5f9f16b566/src/cron.c (L359)

and is basically: sleep until the next minute, then wake up, check the current time, and figure out what needs to be done now. It has explicit handling for various "time jump" scenarios.

@civodul wrote in https://codeberg.org/guile/fibers/pulls/177#issuecomment-13303746: > [...] But then, how did good’ol cron implementations deal with that? Apparently they didn’t: anacron just uses `sleep`: > > [`github.com/cronie-crond/cronie@5f9f16b566/anacron/main.c (L508-L513)`](https://github.com/cronie-crond/cronie/blob/5f9f16b5663becefdd0dd70df31c0ef5ac36f943/anacron/main.c#L508-L513) > > It has a simple heuristic to detect cases where the real time clock jumps forward: > > [`github.com/cronie-crond/cronie@5f9f16b566/anacron/main.c (L391-L396)`](https://github.com/cronie-crond/cronie/blob/5f9f16b5663becefdd0dd70df31c0ef5ac36f943/anacron/main.c#L391-L396) That `sleep` implements a job's optional "delay in minutes", per `anacrontab(5)`. But anacron is intended for jobs of _at most_ daily frequency on systems that are intermittently running. It's part of -- and ordinarily _run by_ -- cron (in cronie). The sleep loop that actually controls the scheduling in cron itself is in `cron.c`'s `main`, starting here: https://github.com/cronie-crond/cronie/blob/5f9f16b5663becefdd0dd70df31c0ef5ac36f943/src/cron.c#L359 and is basically: sleep until the next minute, then wake up, check the current time, and figure out what needs to be done now. It has explicit handling for various "time jump" scenarios.

civodul commented 2026年04月20日 10:05:41 +02:00

@stund Oh thanks, I stand corrected. It’s interesting! And pretty bad too: amazing that the only solution until recently has been to wake up every minute to check the time...

@stund Oh thanks, I stand corrected. It’s interesting! And pretty bad too: amazing that the only solution until recently has been to wake up every minute to check the time...

alexsh commented 2026年04月20日 10:25:17 +02:00

@civodul wrote in #177 (comment):

@stund Oh thanks, I stand corrected. It’s interesting! And pretty bad too: amazing that the only solution until recently has been to wake up every minute to check the time...

I would not call it "pretty bad" — "good enough" is probably more appropriate. At least it's easy to reason about, if you ask me.

@civodul wrote in https://codeberg.org/guile/fibers/pulls/177#issuecomment-13375734: > @stund Oh thanks, I stand corrected. It’s interesting! And pretty bad too: amazing that the only solution until recently has been to wake up every minute to check the time... I would not call it "pretty bad" — "good enough" is probably more appropriate. At least it's easy to reason about, if you ask me.

civodul commented 2026年04月20日 13:56:17 +02:00

@alexsh "Pretty bad" in the sense that waking up every minute to poll the real-time clock is suboptimal (energy-wise, etc.); ideally the program would wake up only when an event occurs.

Which is not incompatible with it being "good enough" in this case. :-)

@alexsh "Pretty bad" in the sense that waking up every minute to poll the real-time clock is suboptimal (energy-wise, etc.); ideally the program would wake up only when an event occurs. Which is not incompatible with it being "good enough" in this case. :-)

alexsh commented 2026年04月20日 14:52:14 +02:00

Sure, but in the face of clock adjustment we are dealing with here, once a minute seems like a reasonable frequency. It's not ideal, but gets the job done.

Also, I don't think there's measurable difference in energy efficiency below 1 Hz. ;)

Sure, but in the face of clock adjustment we are dealing with here, once a minute seems like a reasonable frequency. It's not ideal, but gets the job done. Also, I don't think there's measurable difference in energy efficiency below 1 Hz. ;)

stund commented 2026年04月20日 15:38:29 +02:00

There's a competition here between the classic "Vixie cron" model and Shepherd's approach: for each schedulable job, determine the absolute time of its next occurrence and set a timer to fire at that point.

Shepherd's model seems cleaner, and worked in the face of time jumps partly due to the fact (historical accident?) of Fiber's timers being affected by real time clock changes ... but that causes several other problems, which is why we're here.

But maybe we are "doomed" to re-invent Vixie cron ;)

There's a competition here between the classic "Vixie cron" model and Shepherd's approach: _for each schedulable job, determine the absolute time of its next occurrence and set a timer to fire at that point_. Shepherd's model _seems_ cleaner, and worked in the face of time jumps partly due to the fact (historical accident?) of Fiber's timers being affected by real time clock changes ... but that causes several other problems, which is why we're here. But maybe we are "doomed" to re-invent Vixie cron ;)

This pull request has changes conflicting with the target branch.