Building effective logs in Python

Tracking the chain of actions from a specific point in time can be crucial when debugging and monitoring software operations. That’s why logs were created to record and track events or log information.

Logs are necessary to track errors in software system modules and, if necessary, restore data integrity in information systems. Data logs can be thought of as a method of recording to allow you to find the places where software is running slowly and identify the reason for specific system behavior.

Many programming languages have their own libraries to write logs. The main task of these libraries is a simple, convenient, and effective system for outputting information to a specific source like a file. It’s important to have certain levels of tracking logs based on unique characteristics to process large amounts of information in output logs.

In this post, we are going to walk you through working with logs using the Python language as an example. Let’s get to it!

Why do you need to create logs?

Let’s begin by examining why logs are important in the development and maintenance of information systems, hardware systems, and software systems:

• Help you track and detect random or rare errors, recovering action sequences that led to a specific error.
• Monitor system operations to improve security and identify unusual user behavior of trying to gain unauthorized access to the system.
• Identify weaknesses in infrastructure and cloud services for a stable system operation and to upgrade to more powerful instances, if necessary.
• Analyze the main functionality that is most often used and plan improvements for the most popular functions to make them more convenient.
• Help recover information if the system’s integrity is violated or jeopardized for individual users.
• Help analyze user behavior and improve UI/UX and business logic.

Why shouldn't you use the normal output to display this information, say with print? Because in this case, it will not be possible to write to various files or set the correct levels of log values, which can lead the console output of the program to be confused with the output of logs.

In general, the construction logic for your system logs may look like this.

Image source

Types of log messages

Each log can refer to specific levels (there can be more or fewer levels, depending on your system’s needs). Here are the types of log messages:

• Fatal
• Critical
• Error
• Warning
• Notice
• Info
• Debug
• Trace

These log types help you analyze the level of output of service information, which is necessary to solve specific tasks. Without log types, finding the required entry in the output file would be very difficult.

Creating logs in Python

Overall, logging in Python is very simple. To do so, you need to connect a logging library such as logging and draw an output using the logging.warning () operator.

import logging
logging.warning('This is the warning logging example')

As a result, you will receive a message like the following one in the console: 

WARNING:root:This is the warning logging example

In Python, documentation logs can be created with the following levels:
 

Oftentimes, the size of logs will be larger than the size of your text console and it’s not always convenient to scroll through the logs. In this scenario, you could write logs to a file.

import logging

logging.basicConfig(filename='test.log', level=logging.DEBUG)
logging.debug('This message will be in the log file')

In the file test.log you will have a new line like this:

DEBUG:root:This message will be in the log file

Every time you run this Python script log, text will be added to the existing file, i.e. previous logs lines will be kept.

In the configuration function, you can set another level=logging.DEBUG for the necessary level of logs (INFO, WARNING, ERROR, CRITICAL).

Exception logging

Typically, you need to have additional information when a handled exception is thrown. It’s not easy to track the output when exceptions are thrown in large projects. Here, it’s inappropriate to build a separate output of additional information. Instead, you can successfully use existing loggers in the messages to display information about an exception that has occurred.

import logging

logging.basicConfig(filename='test.log', level=logging.ERROR)
logging.debug('This message will be in the log file')

try:
    print ("Test")
    a = 4 / 0
except ZeroDivisionError as e:
    print ("Division by zero detected")
    logging.error(e, exc_info=True)

Next, the test.log file will look like this

ERROR:root:division by zero
Traceback (most recent call last):
  File "log.py", line 8, in <module>
    a = 4 / 0
ZeroDivisionError: division by zero

Set exc_info=True to show traceback, which is a very helpful option.

Using JSON format as a log output

When your system writes hundreds of thousands of logs per hour, analyzing the necessary entries in the log file becomes nearly impossible. Fortunately, if you organize log outputs in JSON format, you can easily analyze them using special tools to read the values from a specifically generated system log into the required fields.

For example, you can use a library such as json-logging. You can install this library with a simple command using pip:

pip install json-logging

Moreover, as indicated in the documentation for the json_logging library, you can connect the JSON logger using the line logger = logging.getLogger ("test-logger").

Before doing so, be sure to enable the JSON logger using one line: json_logging.init_non_web(enable_json=True).

This is particularly helpful since you can switch between the regular and JSON logger without changing anything else in the whole code for log outputs.

Also, this is a very good way to switch the log format as it preserves the universality principle of the Python code.

import json_logging, logging, sys

# log is initialized without a web framework name
json_logging.init_non_web(enable_json=True)

logger = logging.getLogger("test-logger")
logger.setLevel(logging.DEBUG)
logger.addHandler(logging.StreamHandler(sys.stdout))

logger.info("test logging statement")

Since you now have enable_json = True, the output will be as follows

{"written_at": "2021-02-06T12:15:08.824Z", "written_ts": 1612613708824719000, "msg": "test logging statement", "type": "log", "logger": "test-logger", "thread": "MainThread", "level": "INFO", "module": "jsonlogs", "line_no": 10}

Comment one line json_logging.init_non_web(enable_json=True) and you will switch to simple log format like this:

test logging statement

In addition, there are methods on how to add custom formatters and loggers. You can learn more from the json-logging-python and logging documentation.

Conclusion

In summary, logging is a very useful feature that doesn’t require a lot of effort to implement. With the help of simple actions, very early on in the process of writing code, you can insert the necessary lines to log critical parameters. That way, during system operation, developers of new functionalities and system admins will be very grateful that you implemented logging.

Very rarely does one really look through logs in a corporate system with their eyes. That’s why there are graphical systems to process logs so you can analyze a specifically identified item in text mode.

With minimal effort when writing code, our Svitla System developers employ logging systems to improve the quality of information systems that are developed for the customer. You can always contact Svitla Systems for advice and place orders to develop your projects.