homepage

I want the Python executable to have command line flags which allow
simple configuration of the logging module. Use cases are to run
applications/scripts (which use libraries that use logging calls) with
different logging output without having to edit (ugly) logging config files.
The attached patch demonstrates the idea; it allows to run 'python -l
<options> ...' and passes <options> to a new function
logging._parse_python_options(<options>).

I get the idea. The Python part of the patch demonstrates what you're
getting at, though it can't be used as is - for example the
getattr(logging, a, a) could lead to problems. However a more
intelligent parser (which looked for specific keywords recognised by
basicConfig(), and got the correct values accordingly) wouldn't be much
more complicated. (I'll look at enhancing this part.)
As for the changes to main.c - I am a C/C++ developer but have not made
any changes to Python C code so far - it would be good if a more
experienced committer reviews this part (not sure who - can someone
please reassign/add to nosy list)? Thanks.

> I get the idea. The Python part of the patch demonstrates what you're
> getting at, though it can't be used as is - for example the
> getattr(logging, a, a) could lead to problems. However a more
> intelligent parser (which looked for specific keywords recognised by
> basicConfig(), and got the correct values accordingly) wouldn't be much
> more complicated. (I'll look at enhancing this part.)
> 
> As for the changes to main.c - I am a C/C++ developer but have not made
> any changes to Python C code so far - it would be good if a more
> experienced committer reviews this part (not sure who - can someone
> please reassign/add to nosy list)? Thanks.
Both parts of the patch are only thought to demonstrate the idea.
You said you'll attack the Python part - good.
For the C part, the most prominemt things that are missing are these:
- free(logopt) should be called at the end of the 'if (logopts != NULL) {' block.
- error handling should be improved. Errors in this block should probably exit Python.

If you want to add flags to the main executable, it deserves a
discussion on python-dev IMO.
It could be obtained through an environment variable, e.g.
PYLOGGING_CONFIG; which has the nice side-effect of working for
executable scripts too.

The C code looks basically okay to me. I'd probably use strdup() instead
of the malloc/strlen/strcpy calls. And as Thomas said, the cleanup needs
a little bit of work.

Also possibly relevant was the recent python-dev discussion about
replicating (some of) the command line argument parsing in python so
that it can be used by things other than the 'python' command:
http://mail.python.org/pipermail/python-dev/2009-August/091484.html 

Also, don't forget to update the "python -h" help text.

Why does this need to be built into the interpreter? The script / app
should have logging config support.

> Why does this need to be built into the interpreter? The script / app
> should have logging config support.
It does not need to, but it would be nice.
I think the '-l' flag should be similar to the -W flag.
Or consider for example using unittest.main() as script/app.

If we do include interpreter support, there should be an option to
invoke a configuration file, too:
-l config=path
Mutually exclusive with all the other options. So, you can either use it
to invoke basicConfig or to invoke an arbitrary configuration in a file.
Opinions?

I think I'm with Michael on this one. I'd rather add logging
configuration to any stdlib modules that support being run directly and
want to support logging.

I don't think this is about logging in stdlib modules that are run
directly. I think this is about a library that contains logging calls
(eg: multiprocessing), and is used in j-random-application, and while
prototyping/debugging the application you want to access that logging
information without having to write a logging infrastructure into your
app first. Or you want to access it while debugging someone _else_'s
application...

How about putting this into the logging module instead? Instead of
"python <logging options> <program> <program options>", making it
"python -m logging <logging options> <program> <program options".
This:
 * involves no changes to the core interpreter
 * only requires Python to be written, not C
 * Lets other VMs benefit from the feature immediately
 * Follows what seems to be the emerging convention for this kind of
(eg pdb, unittest)
 * Still allows logging to be configured for arbitrary programs that
wouldn't otherwise be configurable in this way

How do these "global" settings (either via the interpreter or a wrapper
in the logging module) change what an app might do on its own? IOW, if
my app is already written to configure logging, and someone invokes it
with these other settings, which settings are used?

Both Doug and Jean-Paul have made good points, IMO.

> Jean-Paul Calderone:
> 
> How about putting this into the logging module instead? Instead of
> "python <logging options> <program> <program options>", making it
> "python -m logging <logging options> <program> <program options".
> 
> This:
> 
> * involves no changes to the core interpreter
> * only requires Python to be written, not C
> * Lets other VMs benefit from the feature immediately
> * Follows what seems to be the emerging convention for this kind of
> (eg pdb, unittest)
> * Still allows logging to be configured for arbitrary programs that
> wouldn't otherwise be configurable in this way
This is a very good idea, and fully covers my use case.

> How do these "global" settings (either via the interpreter or a wrapper
> in the logging module) change what an app might do on its own? IOW, if
> my app is already written to configure logging, and someone invokes it
> with these other settings, which settings are used?
IMO the same would happen as if logging.basicConfig() would have been called
followed by calls to the custom configuration.

@theller, I'm not sure what your point is. I'm asking what the defined
behavior is if we provide some sort of global way to run a program with
logging configured, and then that app turns around and tries to
reconfigure it. Should the last one to call the configuration
function(s) win, or the first?
I like the idea of adding this feature to the logging module better than
building it into the interpreter, but I still think it opens up areas
for unexpected behavior, and it would be better to just let each
application set up its own logging.

@theller: Although your use case just covers using basicConfig, I can
just see users expecting the same mechanism to invoke configuration
stored in a logging configuration file, which is why I suggested the
config= variant. However, doughellmann raises the valid point of what
the semantics would be if the program you invoke via logging -m does its
own configuration.
With a simple implementation: If a script calls basicConfig after it has
been invoked with logging -m etc. then the basicConfig call won't do
anything, as if the root logger already has some handlers, the call is
essentially a no-op. If the called script loads a configuration file,
that will overwrite the configuration specified via logging -m.
Either way, it's not consistent IMO - sometimes the logging -m
configuration will seem to win, and other times, the called script's
configuration.
Of course, it could be argued that logging -m is intended for scripts
which don't do explicit configuration - I'm not sure of how strong an
argument this is.
Thinking caps on :-)

Ummm, s/logging -m/-m logging/

I retract this request. It seems the idea is not liked or a solution is
not easy.
(The solution I now use is to start Python from a batch file that parses
some command line flags itself, sets environment variables, and my
sitecustomize.py file configures logging).
Thanks for the comments.

Ok. I think it's a reasonable request, but I can see that implementing
it naively without some further thought will lead to (justifiable)
complaints from some users that the behaviour is contradictory or
unintuitive in terms of which configuration "wins" in the event of
ambiguity.

I think Antoine’s envvar idea was a good one, similar to PYTHONWARNINGS.

I was just thinking that it would be nice if logging could be configured through environment variables. This would make it easier to have Python applications adhere to the Twelve-Factor methodology around application configuration:
http://12factor.net/config
Currently, applications need to handle this themselves, but I wonder if there's value in deciding on a best practice and providing for it at a lower level (the stdlib logging module).

I guess I should be more specific about the 12factor thing I just mentioned, because http://12factor.net/logs says:
>>>
A twelve-factor app never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout. During local development, the developer will view this stream in the foreground of their terminal to observe the app’s behavior.
>>>
The part that I want to be configurable is what the log levels are for various loggers.
For example, in production, you probably want most or all of your loggers set to WARN or such.
When running in development, ideally the developer can turn on more verbose logging without having to edit config files.
e.g.:
This is a Pyramid application, but environment variables of course could be used with any kind of application -- Django, etc.
```
LOGGER_ANWEB_LEVEL=debug pserve app.ini
```