-
Notifications
You must be signed in to change notification settings - Fork 152
logger: apply a log config file to the live logger without a restart - #1791
Open
cosmin-staicu wants to merge 1 commit into
Open
logger: apply a log config file to the live logger without a restart #1791cosmin-staicu wants to merge 1 commit into
cosmin-staicu wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: 907d9f7 The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
cosmin-staicu
force-pushed
the
feat/logger-config-file-watch
branch
from
September 10, 2026 08:33
4451329 to
9d16f84
Compare
...start
The pieces for changing levels at runtime were already here — the root level and
every component level are zap.AtomicLevel, Config.Update pushes new values into
them, and newSharedConfig registers itself as an update observer — but nothing
ever called Update, so a level change meant restarting the process.
Add the missing trigger: when LK_LOG_CONFIG_PATH is set, poll that file and apply
it to the Config the service is holding (LK_LOG_CONFIG_INTERVAL overrides the
30s default). The hook sits in newSharedConfig, which is the one path every
consumer reaches — livekit-server via InitFromConfig, livekit-sip via
NewZapLogger — so no binary needs its own flag or call site.
Polling rather than fsnotify because the target is a mounted ConfigMap: kubelet
swaps the ..data symlink instead of rewriting the file, so a watch on the file
never fires.
The file is a declarative overlay on the startup config, not on whatever was
applied last: WatchConfigFile snapshots the config once at startup and every file
is decoded over that baseline, so a key the file omits falls back to its startup
value and emptying the file to `{}` restores the levels the process booted with.
Decoding over the config in force instead would make an empty file a no-op once
`level: debug` had been applied, and would keep applying a component_levels entry
after it disappeared from the file.
Two details that would otherwise bite:
- Update assigns every field, so a partial file decoded into a zero Config would
silently reset the rest. Config.snapshot copies the data fields and the file is
unmarshalled over that, leaving unspecified keys — including ComponentLevels,
where livekit-server puts pion_level — as they were.
- sharedConfig kept the caller's live *Config and read ComponentLevels from it
under its own mutex, while Update writes those fields under Config.lock. Two
mutexes over the same memory was harmless while Update was unreachable; now it
is reachable, so sharedConfig holds a snapshot it owns instead.
An unreadable file (an optional ConfigMap not yet mounted), unchanged bytes and
malformed YAML all leave the config in force untouched, without logging once per
interval. A file that goes away is deliberately not a reset — a transient read
error would otherwise flap levels on a live process; emptying it to `{}` is the
reset.
Signed-off-by: Cosmin Staicu <cosmin.staicu@uipath.com>
cosmin-staicu
force-pushed
the
feat/logger-config-file-watch
branch
from
September 10, 2026 08:46
9d16f84 to
907d9f7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
Set
LK_LOG_CONFIG_PATHto a YAML file and the logger applies edits to it, so a running process can change log level without a restart.Why
The parts for this are already in
logger: the root level and every component level arezap.AtomicLevel,Config.Updatepushes new values into them, andnewSharedConfigregisters as an update observer. Nothing ever calledUpdate, so a level change still meant restarting the process and losing the state you wanted to look at.What
WatchConfigFilepolls the file (interval fromLK_LOG_CONFIG_INTERVAL, default 30s) and applies it to theConfigthe service holds. The hook sits innewSharedConfig, which every consumer reaches (livekit-server viaInitFromConfig, livekit-sip viaNewZapLogger), so no binary needs its own flag or call site. With the env var unset nothing changes.Polling instead of fsnotify because the target is a mounted ConfigMap: kubelet swaps the
..datasymlink rather than rewriting the file, so a watch on the file never fires.The file is an overlay on the startup config, not on whatever was applied last.
WatchConfigFilesnapshots the config once at startup and decodes every file over that baseline, so a key the file omits falls back to its startup value, and emptying the file to{}restores the levels the process booted with.Worth a look
Config.Updateassigns every field, so a partial file decoded into a zeroConfigwould reset the rest.Config.snapshotcopies the data fields first and the file is unmarshalled over that, which leaves unspecified keys alone, includingComponentLevelswhere livekit-server putspion_level.sharedConfigused to hold the caller's live*Configand readComponentLevelsunder its own mutex whileUpdatewrote those fields underConfig.lock. That was harmless whileUpdatewas unreachable. It is reachable now, sosharedConfigkeeps a snapshot it owns.An unreadable file, unchanged bytes and malformed YAML all leave the config in force alone, without logging once per interval. A file that disappears is not a reset either, since a transient read error would flap levels on a live process. Emptying it to
{}is the reset.Tests in
logger/configwatch_test.go.