Allow formatter customizations #18

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

yyuu merged 8 commits into fluent:master from EvaSDK:allow-formatter-customizations

Jun 3, 2015

Merged

Allow formatter customizations #18

yyuu merged 8 commits into fluent:master from EvaSDK:allow-formatter-customizations

Jun 3, 2015

Conversation

@EvaSDK

Copy link

Contributor

@EvaSDK EvaSDK commented Aug 1, 2014

I am currently pursuing the goal of using fluent just like any other logging Handler/Formatter via logging.config

In order to do that I had to solve a couple of problems that make default FluentRecordFormatter not easy to deal with. It is now able to take a format argument which is a dictionary that holds keys and values. Values are regular LogRecord standard Python %-style mapping keys. The defaults are unchanged.

I also added a couple of fixes I found in other pull requests as I found them as well during my tests, so this should cover Issue #7, PR #8 and PR #16.

Finally I added a sphinx ready docstring to FluentRecordFormatter._structuring although this function could probably be merged in format directly since I don't see it used anywhere else.

Once merged, the following sample program:

import logging
from fluent.handler import FluentHandler, FluentRecordFormatter
logger = logging.getLogger('toto')
logger.setLevel(logging.DEBUG)
handler = FluentHandler('hello.test')
handler.setLevel(logging.DEBUG)
formatter = FluentRecordFormatter(fmt={
 'msg': '%(message)s',
 'level': '%(levelname)s',
 'hostname': '%(hostname)s',
})
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.info('hel""lo')

logs

2014年08月01日 13:59:05 +0200 hello.test: {"msg":"hel\"\"lo","message":"hel\"\"lo","hostname":"seta.local","level":"INFO"}

to fluentd.

@kdeldycke

Copy link

kdeldycke commented Aug 1, 2014

👍

@EvaSDK

Copy link

Contributor Author

EvaSDK commented Aug 1, 2014

A sample configuration would be:

logging:
 version: 1
 formatters:
 brief:
 format: '%(message)s'
 default:
 format: '%(asctime)s %(levelname)-8s %(name)-15s %(message)s'
 datefmt: '%Y-%m-%d %H:%M:%S'
 fluent_fmt:
 '()': fluent.handler.FluentRecordFormatter
 format:
 message: '%(message)s'
 level: '%(levelname)s'
 hostname: '%(hostname)s'
 where: '%(module)s.%(funcName)s'
 handlers:
 console:
 class : logging.StreamHandler
 level: DEBUG
 formatter: default
 stream: ext://sys.stdout
 fluent:
 class: fluent.handler.FluentHandler
 host: localhost
 port: 24224
 tag: test.logging
 formatter: fluent_fmt
 level: DEBUG
 null:
 class: logging.NullHandler
 loggers:
 amqp:
 handlers: [null]
 propagate: False
 conf:
 handlers: [null]
 propagate: False
 '': # root logger
 handlers: [console, fluent]
 level: DEBUG
 propagate: False

loaded likeso:

import logging.config
import yaml
with open('logging.yaml') as fd:
 conf = yaml.load(fd)
logging.config.dictConfig(conf['logging'])

the way to customize formatter via logging.config.dictConfig drove me a bit crazy until I found the proper documentation. So I figured it would be useful to add it as an example here.

@EvaSDK EvaSDK mentioned this pull request

Aug 5, 2014

Add formatted exception #19

Open

@EvaSDK EvaSDK force-pushed the allow-formatter-customizations branch from 5082d68 to fbd05d8 Compare

February 3, 2015 14:49

@EvaSDK

Copy link

Contributor Author

EvaSDK commented Feb 3, 2015

Branch updated against 0.3.5.

@kiyoto

Copy link

Contributor

kiyoto commented Feb 3, 2015

@EvaSDK

The idea looks good to me. Do you think you can add a test for the string input case? After that it's ready to be merged.

@EvaSDK

Copy link

Contributor Author

EvaSDK commented Feb 4, 2015

I can look into it yes.

@ngiallelis ngiallelis mentioned this pull request

Feb 15, 2015

Make FluentRecordFormatter functional for custom formats #31

Closed

@EvaSDK EvaSDK force-pushed the allow-formatter-customizations branch from fbd05d8 to 2768952 Compare

February 25, 2015 15:03

@EvaSDK

Copy link

Contributor Author

EvaSDK commented Feb 25, 2015

Added some tests and fixed a few issues while doing so.

The most annoying thing to me is the drop of python 2.6 support due to class changes in python logging module and how it handles the asctime attribute. I think dropping 2.6 support is not too bad anyway due to it being unsupported on most distributions nowadays. If you still want to support it, I can elaborate an alternative solution.

EvaSDK and others added 8 commits

May 27, 2015 15:34

@EvaSDK


 Add developement marker in version

3c8fc0d

@EvaSDK


 Add support for customizing structured message

d45e765

Should solve concerns from issue fluent#7.

@EvaSDK


 Use basestring instead of str, like in FluentHandler._add_dic

130be83

For consistency's sake.

@EvaSDK


 Add compatibility with regular (non structured) logging usage

1cde07a

Part of fluent#8.

@EvaSDK


 Add docstring to FluentHandler._structuring

d08a557

@EvaSDK


 Fix support for asctime attribute

6cfd06a

This does not work in python2.6 as it did not use usesTime to detect the
presence of asctime in fmt.

@EvaSDK


 Add test for logging json encoded strings

8cd0068

@quasipedia @EvaSDK


 Update README to proper use of FluentRecordFormatter

70d68cb

@EvaSDK EvaSDK force-pushed the allow-formatter-customizations branch from 2768952 to 70d68cb Compare

May 27, 2015 13:36

@EvaSDK

Copy link

Contributor Author

EvaSDK commented May 27, 2015

Added py26 compatibility back. It does not look pretty but if you want it for older distributions, you have it.
I included README update from PR #22.

This fixes: issue #7, PR #8, PR #22 / issue #23

With PR #31, this adds up to 3 pull requests to fix conformance to logging classes. Please tell me what to do with this PR, it has been almost a year and I did everything I could to satisfy your modification requests.

@EvaSDK EvaSDK mentioned this pull request

May 27, 2015

logging handler yields 'record' as string instead of map #34

Closed

@repeatedly

Copy link

Member

repeatedly commented May 27, 2015

@yyuu Could you check this PR?
I confirmed this change resolves #34 issue.
We should fix issue 34 to avoid logging trouble.

@yyuu

Copy link

Member

yyuu commented May 29, 2015

Looks good to me.

@repeatedly

Copy link

Member

repeatedly commented May 30, 2015

@yyuu Feel free to merge this PR and release new version. Maybe update minor version is better?
After that, we will close the issue on fluentd repository.

@EvaSDK

Copy link

Contributor Author

EvaSDK commented May 30, 2015

The PR contains a version bump to 0.4.0 as I felt this was an important enough change.

@repeatedly

Copy link

Member

repeatedly commented May 31, 2015

How about this PR? > #31

@yyuu

Copy link

Member

yyuu commented Jun 1, 2015

@repeatedly I don't have permission to upload fluent-logger to PyPI....

@EvaSDK

Copy link

Contributor Author

EvaSDK commented Jun 1, 2015

#31 squashed all my commits in one and adds a feature that is debatable on its own. Imho, it should be reworked as a single commit on top of my branch first.

[画像:yyuu]

yyuu reviewed

Jun 3, 2015

View reviewed changes

.travis.yml

Copy link

Member

@yyuu yyuu Jun 3, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@EvaSDK Why did you remove 2.6 from .travis.yml? It's still in tox.ini and seems working fine at least for me.

Copy link

Contributor Author

@EvaSDK EvaSDK Jun 3, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like I forgot to update .travis.yml when I re-added support for it. I'll fix that.

Copy link

Member

@yyuu yyuu Jun 3, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, got it. I'll fix it by myself.

@yyuu yyuu merged commit 70d68cb into fluent:master

Jun 3, 2015

yyuu added a commit that referenced this pull request

Jun 3, 2015

@yyuu


 Revert CI configuration for py26 ( #18 )

39187c6

@EvaSDK

Copy link

Contributor Author

EvaSDK commented Jun 3, 2015

w00t, thanks.

@EvaSDK EvaSDK deleted the allow-formatter-customizations branch

June 17, 2015 08:29

This was referenced Jun 17, 2015

When an ordinary character string is received. #8

Closed

Bugfix/py3 recordformatter incompatibility #22

Closed

@ngiallelis ngiallelis mentioned this pull request

Jul 2, 2015

Update Readme, including custom formatter configuration for handler #39

Merged

Labels

None yet

6 participants

@EvaSDK @kdeldycke @kiyoto @repeatedly @yyuu @quasipedia

Allow formatter customizations #18

Allow formatter customizations #18

Uh oh!

Conversation

@EvaSDK EvaSDK commented Aug 1, 2014

Uh oh!

kdeldycke commented Aug 1, 2014

Uh oh!

EvaSDK commented Aug 1, 2014

Uh oh!

EvaSDK commented Feb 3, 2015

Uh oh!

kiyoto commented Feb 3, 2015

Uh oh!

EvaSDK commented Feb 4, 2015

Uh oh!

EvaSDK commented Feb 25, 2015

Uh oh!

EvaSDK commented May 27, 2015

Uh oh!

repeatedly commented May 27, 2015

Uh oh!

yyuu commented May 29, 2015

Uh oh!

repeatedly commented May 30, 2015

Uh oh!

EvaSDK commented May 30, 2015

Uh oh!

repeatedly commented May 31, 2015

Uh oh!

yyuu commented Jun 1, 2015

Uh oh!

EvaSDK commented Jun 1, 2015

Uh oh!

@yyuu yyuu Jun 3, 2015

Choose a reason for hiding this comment

Uh oh!

@EvaSDK EvaSDK Jun 3, 2015

Choose a reason for hiding this comment

Uh oh!

@yyuu yyuu Jun 3, 2015

Choose a reason for hiding this comment

Uh oh!

EvaSDK commented Jun 3, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants