Python logging with the standard library

Posted on

Let’s condense the basics of the logging Python module.

tldr

  • If you don’t configure any logger, only the log levels WARN and up will show up.
  • In your program (not a library), set up the root looger as one of the first things you do. Calling logging.basicConfig does that for example.
  • In each module, have a global logger = loging.getLogger(__name__) variable.
  • If you write a library, never add a handler to your loggers, because this is the consumer’s job. Either don’t add any handlers, or use a NullHandler - see here

Logging module basics

In a nutshell:

  • A logger can have:

    • handlers: a log handler is responible for the output of a log record. A logger can have several handlers: e.g., it can log to console and a file. Each handler can be configured differently, e.g. regarding the log level. If you define a custom handler, you need to implement the emit(record) method that outputs a log record.
    • formatters: a handler can have a formatter that defines how a log record should be formatted.
    • filters: a handler (or a logger) can filter messages and only emits log records that pass the filter. You can simply add a filter function that takes a record (string) and returns zero or one.

  • Loggers follow a hierachy. When a child logger receives a log record it first passes it to its own handlers. Per default, log records are then passed to any parent loggers. This is important for configuring the loggers. The hierarchy can be established in two ways:

    1. By configuring the root logger. This is done by e.g. calling logging.basicConfig. Any logger that you create with logging.getLogger('my_name') will be a child of the root logger. Note: calling logging.basicConfig if the root logger is already configured will have no effect.
    2. By establishing a logger hierarchy via dot notation: main.child, where the main logger would the parent of the child logger. This way, we could define a main logger in our main module and configure it there: main_logger = logging.getLogger("main") And in each submodule we explicitly create a child logger: library_logger = logging.getLogger("main.library"). Now, in tutorials and the documentation you always see the recommendation to use logging.getLogger(__name__). This makes a lot of sense, if you have a package structure, because then the logging hierarchy is automatically configured. In an entry module, I nowadays always create a logger via logger = logging.getLogger(__name__)

Tidbits

  • You can pass additional information to logging calls. See LoggerAdapters and the extra keyword that allows to use additional parameters in the format string.
  • In larger projects, configure loggers via text configuration. See the logging.config documentation. Either use logging.config.fileConfig (which uses the configparser module) with ini-style config files, or use a library to de-serialize a text format to a dictionary and then initialize the logger with logging.config.dictConfig.

Resources