Solving Airflow - ImportError: Unable to load custom logging from log_config.DEFAULT_LOGGING_CONFIG

I was working in Airflow and at the moment that I tried to configure a custom log I got the following error:

ImportError: Unable to load custom logging from 
airflow.config.log_config.LOGGING_CONFIG due to 
section/key [logging/logging_level] not found in config

As 99% of the normal people, I went in Stack Overflow and checked the answer given by Meny Issakov.

Solution

So based on its response, I got the following (working) solution doing the following steps:

1) I opened the file airflow.cfg

2) I’ve Iinclude a new section in the file, below the [core] section, called [logging] using the following code: [logging] logging_config_class = log_config.DEFAULT_LOGGING_CONFIG

3) I restarted the scheduler

However, going a bit into the root cause of the problem, I got a (non-definitive) conclusion.

ELI5 the reason of problem

The file airflow.cfg is missing a section called [logging].

Why the problem happened?

At the time that the scheduler starts, it accesses the [core] section in the airflow.cfg and search the logging path.

And the information where the logs will be stored it’s found in logging_config_class parameter.

However, even if we put logging_config_class = log_config.DEFAULT_LOGGING_CONFIG in the [core] section, the scheduler it’s not gonna work either.

Why? It’s because there’s a mismatch between the log_config.py logging handlers and with the airflow.cfg.

In the logging handlers in the log_config.py they have the following conf.get to get the logging configurations:

LOG_LEVEL: str = conf.get('logging', 'LOGGING_LEVEL').upper()  
LOG_FORMAT: str = conf.get('logging', 'LOG_FORMAT')

The first parameter it’s the section that will be scanned in the airflow.cfg file, but by default, there’s no section called [logging] in the airflow.cfg file, and this causes the following error in the scheduler initialization:

ImportError: Unable to load custom logging from log_config.DEFAULT_LOGGING_CONFIG due to section/key [logging/fab_logging_level] not found in config

I hope it helps.

PS 1: By the way, I got the same error when I was in the excellent Udemy course provided by Marc Lamberti specifically in the Section 8: Monitoring Apache Airflow Lecture - Practice - Setting up custom logging

PS 2: The log config has the following format in the current version of Airflow that I’m using (1.10.14):

PS 3: The modified version of airflow.cfg file with the [logging] section is: