-
Notifications
You must be signed in to change notification settings - Fork 3.5k
Feat/temperature scaling confidence calibration #1434
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
David-Magdy
wants to merge
2
commits into
JaidedAI:master
from
David-Magdy:feat/temperature-scaling-confidence
Open
Feat/temperature scaling confidence calibration #1434
David-Magdy
wants to merge
2
commits into
JaidedAI:master
from
David-Magdy:feat/temperature-scaling-confidence
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Feat: added temperature scaling to recognition and replace custom confidence with average probability - Introduced `temperature` parameter in `recognizer_predict` and `get_text` to calibrate model confidence output. - Applied temperature scaling to logits before softmax to soften or sharpen confidence. - Swapped `custom_mean` (geometric-inspired) for simple mean of max probabilities, to yield more interpretable confidence scores.
Aligned API function definitions with the new temperature scaling feature.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add temperature scaling for confidence calibration and simplify confidence metric
Description
This PR introduces a
temperature
parameter to the recognition pipeline, allowing users to calibrate model confidence, softening overconfident models or boosting underconfident ones. It applies temperature scaling to logits before the softmax layer inrecognizer_predict
, and replaces the geometric-root-basedcustom_mean
with a standard average of token-level max probabilities, making confidence scores more interpretable.Why this matters
Confidence calibration makes EasyOCR more trustworthy in edge cases, especially for challenging scripts like Arabic. This tweak gives fine-grained control over confidence behavior across model variants or data types, without changing the core model architecture. The code remains purely additive, optional, and fully backward-compatible.
Changes
temperature
parameter (default = 1.0) in:recognizer_predict
get_text
custom_mean
confidence calculation with standard average of max probabilities.temperature
parameter.Example: Reducing overconfidence with temperature scaling
We tested the new
temperature
parameter on Arabic OCR.The baseline model was highly overconfident, often assigning >0.95 confidence to incorrect predictions.
Applying temperature scaling with
temperature=1.5
reduced these inflated scores, producing more realistic confidence estimates.Before (temperature = 1.0, default):
After (temperature = 1.5):
This demonstrates how temperature scaling can make EasyOCR confidence scores more trustworthy in practice.
Testing
get_text
withtemperature=1.5
produces expected reduction in average confidence scores.temperature=1.0
.Backward compatibility
temperature
defaults to1.0
, so existing users experience no behavior change unless they explicitly set it.Next steps
temperature
parameter.Maintainers:
This PR is self-contained and backward compatible. The included example shows one practical case (reducing overconfidence). Further use cases like boosting confidence can be demonstrated in follow-up tests if needed.