This repository was archived by the owner on Oct 23, 2023. It is now read-only.
-
-
Notifications
You must be signed in to change notification settings - Fork 656
Fix celery error logging #1267
Open
Open
Fix celery error logging #1267
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
@tilboerner
tilboerner
force-pushed
the
fix-celery-error-logging
branch
2 times, most recently
from
July 2, 2018 15:47
99e3a8e to
e577bcf
Compare
@tilboerner thx! Mind rebasing with master to get the tests passing.
@tilboerner
tilboerner
force-pushed
the
fix-celery-error-logging
branch
from
July 13, 2018 15:31
e577bcf to
e380432
Compare
@ashwoods Rebased it. Could you restart that one failed travis job? Looks like some pip download timed out.
blodone
commented
Jul 19, 2018
its really really painful to enable logging celery to sentry... +1
huangsam
commented
Sep 12, 2018
Is there anything blocking this from getting released? This PR would be incredibly useful for folks who need to track Celery errors via the Raven client.
To all, I think the proper fix is to upgrade to the new Python SDK which bypasses those problems entirely.
I'm a bit vary of fixing this in raven as it is a breaking change for something that is effectively in maintenance mode.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
Error logging from Celery tasks is broken out of the box. While uncaught exception reporting works, error-level log messages from inside Celery worker do not get reported to Sentry.
For the user, it's not obvious that this is happening, or what can be done to fix it.The setup instructions for Django and Celery are not sufficient to get it working. There is documentation on raven.readthedocs.io suggesting that the Django handler "captures the errors" from workers. Consequently, there is #922, suggesting a workaround.
The problem is that when setting up logging for worker processes, Celery configures the task logger to not propagate to the root logger:
https://github.com/celery/celery/blob/47ca2b462f22a8d48ed8d80c2f9bf8b9dc4a4de6/celery/app/log.py#L132
https://github.com/celery/celery/blob/47ca2b462f22a8d48ed8d80c2f9bf8b9dc4a4de6/celery/app/log.py#L176
This prevents log events from reaching the Celery handler in the root logger. (Note that this setup does not happen in the Django shell, for example. There, things work fine.)
The workaround in #922 disables this logging setup completely. I'd argue that it's cleaner to work with whatever Celery decides to do and not mess with the logging system any further, beyond adding sentry handlers.
So, this PR adds a signal handler for the task logger, to give it the sentry handler if needed. It also adds code so that the
DjangoCeleryHandlersets this up properly out of the box, if any of theCELERY_LOGLEVELsettings are defined. (This way it's opt-in; it could also be enabled by default.)People should not have to discover the hard way that their errors do not properly get logged, so I think Raven should really fix this and work out of the box. If any manual action remains to be required, this should be noted in the documentation.
If you're willing to consider this PR, I'd be happy to add tests and documentation. In that case, it would be great if you could point me towards the proper places to put them.