-
Notifications
You must be signed in to change notification settings - Fork 445
Comments
Add env variables to configure manifest cache#2993
Open
kris-gaudel wants to merge 5 commits intoapache:main from
Open
Add env variables to configure manifest cache #2993kris-gaudel wants to merge 5 commits intoapache:main from
kris-gaudel wants to merge 5 commits intoapache:main from
Conversation
Contributor
Author
@kevinjqliu @Fokko @rambleraptor Please lmk what you think!
thomas-pfeiffer
commented
Feb 18, 2026
@kris-gaudel I have a few questions:
- Would it make sense to list the new configuration parameters somewhere in
mkdocs/docs/configuration.md, so that it shows up in the documentation as well? - Would it make sense to allow
PYICEBERG_MANIFEST__CACHE__SIZEto be set to0to effectively disable the cache?- It's mentioned in Deprecate support for
cache(self)returningNone. Why? tkem/cachetools#352 (comment) thatmaxsize = 0can be used to bypass the cache. - This would make
PYICEBERG_MANIFEST__CACHE__ENABLEDobsolete, right?
- It's mentioned in Deprecate support for
- What's the added benefit of having a
_ManifestCacheManagerclass?- Wouldn't
_manifest_cache: LRUCache[str, ManifestFile] = LRUCache(maxsize=int(environ.get("PYICEBERG_MANIFEST__CACHE__SIZE", 128)))do the trick as well? (obviously an oversimplified example)
- Wouldn't
Fokko
Fokko
reviewed
Feb 20, 2026
Contributor
@Fokko
Fokko
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @kris-gaudel for raising this. I think it is reasonable to make this configurable. But I think we should reduce the number of changes and simplify where possible. For example, do we need PYICEBERG_MANIFEST__CACHE__ENABLED or can we set PYICEBERG_MANIFEST__CACHE__SIZE to 0 instead.
@kris-gaudel
kris-gaudel
force-pushed
the
kris-gaudel/configurable-manifest-cache
branch
from
February 21, 2026 23:18
9ceb651 to
4cad1c3
Compare
@kris-gaudel
kris-gaudel
force-pushed
the
kris-gaudel/configurable-manifest-cache
branch
from
February 22, 2026 00:34
f3b9f3d to
f411547
Compare
geruh
geruh
reviewed
Feb 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
Closes #2952
Rationale for this change
Through discussion in issue #2325, we realized that there was a memory leak in the manifest cache. PR #2951 fixed this memory leak, but we decided that it would be best for developer experience if we developers could configure the cache for their needs.
This includes the ability to disable/enable the manifest cache, and configure its size
Are these changes tested?
Yes - unit tests are included to verify these changes
Are there any user-facing changes?
Yes:
PYICEBERG_MANIFEST__CACHE__SIZEcan be used to configure the size of the manifest cache