Difference between interaction log and "debug" or "application" log

Question

Let's say I need two logs coming out of my application. One is for debugging issues and making sure the application runs correctly and the other is used for user statistics and general analysis of the entire interaction. The latter should include enough information to spot bottlenecks and the entire behavior per user.

First of all, how would you call each log? Are they both called a "log"? Is one an audit log? What's the correct professional terminology for each?

Second of all, how would you design the statistics log?

In Python for example, I can use the logging module to insert random logging.info or logging.debug calls for the application log, quite about anywhere, and even if some information will be missed, it's not an issue.

The statistics log though needs to keep a unique session id for example, all throughout the program. This causes design issues and forces me to either put it globally per thread or pass it around just about everywhere, even for entirely decoupled utility modules, and thus ruins the architectural layers. I might need raw data from the sockets for the statistics, and I might need high application-tier data too, wrecking quite a havoc as of where to put the logging lines, where to put some per-request identifiers like session id, and whether some layers or decoupled modules should even be aware of the request (in case I need some internal statistics showing the operation of the decoupled modules under each request). Statistic log should also be machine parsable ofc, so would you even use the python logging module for that?

Bottom line, is there any general idiom to design those logs?

• Thomas Junk: You want to take a look at github.com/etsy/statsd , appdynamics.com/python , newrelic.com/python
• Bharel: @ThomasJunk I'm more interested in how to design such a thing with respect to cross-tier uses.
• Thomas Junk: hilton.org.uk/blog/microservices-correlation-id
• Bharel: @ThomasJunk that's much closer to what I was looking for. Thank you. Question is, if I have a utility module that accepts a request and returns an appropriate answer, let's say that it returns 1 if the header contains banana or 0 if it doesn't. Inside the utility module I want it to log "found banana in request, returning 1". It means the utility moudle will have to be aware of the correlation ID and every module in the whole code will need to pass it around. Second of all, would you put that same log which is used for statistics together with random debug messages such as "system is up"?
• Thomas Junk: The correlationID is only a "transactional" identifier to mark things which could be grouped together. The question according banana is in the request or not is independend from the id. Each service would log separately and with the correlation id, you would find, which things belong together.

Answer 1

I would call them

• myApplication.technical.log for real technical issues, errors etc
• myApplication.statistics.log for your mentioned statistics
• myApplication.user.log for user related interactions

and so on, depending on the concrete usage scenario.

• Thomas Junk: This might be a first step in the right direction.
• Bharel: And would you use python's logging module for each and later on parse it for statistics?

Answer 2

What you're describing sounds like TRACE and DEBUG logs.

TRACE

These are typically used to log execution paths through the code.

DEBUG

These logs are used to capture more in depth information like the value of certain variables. Such information can be useful as diagnostic information in the event of a crash or error.

---

I'm not too familiar with Python, but any logging framework worth its salt will provide support for these out of the box along with other logging levels such as WARNING, ERROR and CRITICAL.

For gathering the stats themselves, it is preferable in my view to keep the raw logs as they are and then have another process to roll up how many of each type of error you're getting over a certain time frame. It could in theory go in the code, but this generally, is orthogonal to the purposes of the application so it should ideally be kept as a separate piece. The requirements for such stats tend to change over time so pre-aggregating information at the expense of keeping the raw logs is rarely beneficial in the long term IMHO YMMV.

As far as a general idiom goes, it should be possible to increase or decrease the amount of logging the code produces without having to edit the code i.e. by changing the logging level.

• Thomas Junk: I think TO is looking for "monitoring", which would be some kind of "TRACE" always ;)
• Bharel: Both trace and debug would be what I call the "application" log. I'm interesting in another log used for tracking user interaction that will be automatically parsed, i.e. "how many users passed a certian criteria", "what was the most frequent visited page", "which user received what response" and then make computational analysis on it
• Robbie Dee: @Bharel So just use another logging level to do whatever you need...
• Bharel: @RobbieDee different levels or an entirely different log? Would you log "user x requested y" with lots of extra request information together with "server started"? Logging them together would require you to filter the statistics data out quite heavily. Maybe putting the data in a database would fit better? How does Google for example store the user interaction separately from server messages?
• Robbie Dee: If the volume of logs you're producing is quite high, you could simply direct different levels to different files. This is useful when new systems go live. Initially you're only interested in ERROR and CRITICAL. Later on you might want to peruse the WARNING log when you're polishing the code further.