Skip to content

Navigation Menu

Sign in
Sign up

logger: apply a log config file to the live logger without a restart - #1791

Open
cosmin-staicu wants to merge 1 commit into
livekit:main from
UiPath:feat/logger-config-file-watch
Open

logger: apply a log config file to the live logger without a restart #1791
cosmin-staicu wants to merge 1 commit into
livekit:main from
UiPath:feat/logger-config-file-watch

Conversation

@cosmin-staicu

@cosmin-staicu cosmin-staicu commented Sep 10, 2026
edited
Loading

Copy link
Copy Markdown

Set LK_LOG_CONFIG_PATH to a YAML file and the logger applies edits to it, so a running process can change log level without a restart.

Why

The parts for this are already in logger: the root level and every component level are zap.AtomicLevel, Config.Update pushes new values into them, and newSharedConfig registers as an update observer. Nothing ever called Update, so a level change still meant restarting the process and losing the state you wanted to look at.

What

WatchConfigFile polls the file (interval from LK_LOG_CONFIG_INTERVAL, default 30s) and applies it to the Config the service holds. The hook sits in newSharedConfig, which every consumer reaches (livekit-server via InitFromConfig, livekit-sip via NewZapLogger), so no binary needs its own flag or call site. With the env var unset nothing changes.

Polling instead of fsnotify because the target is a mounted ConfigMap: kubelet swaps the ..data symlink rather than rewriting the file, so a watch on the file never fires.

The file is an overlay on the startup config, not on whatever was applied last. WatchConfigFile snapshots the config once at startup and decodes every file over that baseline, so a key the file omits falls back to its startup value, and emptying the file to {} restores the levels the process booted with.

Worth a look

Config.Update assigns every field, so a partial file decoded into a zero Config would reset the rest. Config.snapshot copies the data fields first and the file is unmarshalled over that, which leaves unspecified keys alone, including ComponentLevels where livekit-server puts pion_level.

sharedConfig used to hold the caller's live *Config and read ComponentLevels under its own mutex while Update wrote those fields under Config.lock. That was harmless while Update was unreachable. It is reachable now, so sharedConfig keeps a snapshot it owns.

An unreadable file, unchanged bytes and malformed YAML all leave the config in force alone, without logging once per interval. A file that disappears is not a reset either, since a transient read error would flap levels on a live process. Emptying it to {} is the reset.

Tests in logger/configwatch_test.go.

changeset-bot Bot commented Sep 10, 2026
edited
Loading

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 907d9f7

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages
Name Type
github.com/livekit/protocol Patch
@livekit/protocol Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

...start
The pieces for changing levels at runtime were already here — the root level and
every component level are zap.AtomicLevel, Config.Update pushes new values into
them, and newSharedConfig registers itself as an update observer — but nothing
ever called Update, so a level change meant restarting the process.
Add the missing trigger: when LK_LOG_CONFIG_PATH is set, poll that file and apply
it to the Config the service is holding (LK_LOG_CONFIG_INTERVAL overrides the
30s default). The hook sits in newSharedConfig, which is the one path every
consumer reaches — livekit-server via InitFromConfig, livekit-sip via
NewZapLogger — so no binary needs its own flag or call site.
Polling rather than fsnotify because the target is a mounted ConfigMap: kubelet
swaps the ..data symlink instead of rewriting the file, so a watch on the file
never fires.
The file is a declarative overlay on the startup config, not on whatever was
applied last: WatchConfigFile snapshots the config once at startup and every file
is decoded over that baseline, so a key the file omits falls back to its startup
value and emptying the file to `{}` restores the levels the process booted with.
Decoding over the config in force instead would make an empty file a no-op once
`level: debug` had been applied, and would keep applying a component_levels entry
after it disappeared from the file.
Two details that would otherwise bite:
- Update assigns every field, so a partial file decoded into a zero Config would
 silently reset the rest. Config.snapshot copies the data fields and the file is
 unmarshalled over that, leaving unspecified keys — including ComponentLevels,
 where livekit-server puts pion_level — as they were.
- sharedConfig kept the caller's live *Config and read ComponentLevels from it
 under its own mutex, while Update writes those fields under Config.lock. Two
 mutexes over the same memory was harmless while Update was unreachable; now it
 is reachable, so sharedConfig holds a snapshot it owns instead.
An unreadable file (an optional ConfigMap not yet mounted), unchanged bytes and
malformed YAML all leave the config in force untouched, without logging once per
interval. A file that goes away is deliberately not a reset — a transient read
error would otherwise flap levels on a live process; emptying it to `{}` is the
reset.
Signed-off-by: Cosmin Staicu <cosmin.staicu@uipath.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

AltStyle によって変換されたページ (->オリジナル) /