Kafka log4j file maintenance

July 13, 2015 - BigData

By default, the Kafka 2.8.0 log directory gets populated with rolling hourly files. For significant real-time applications, the server-side log files contained within this directory can quickly fill up an entire disk.

The log4j configuration file ($KAFKA_HOME/config/log4j.properties) uses theĀ DailyRollingFileAppender to generate log files. Unfortunately, this particular appender does not have a clean-up mode that can allow for the deletion of older files. Thus, we need to set up a separate and recurring clean operation.

Here’s how to do that:

Write a crontab entry

0 0 * * * find $KAFKA_HOME/logs -type f -mtime +7 -exec rm -f {} \;

Explanation:

  • “0 0 * * *” tells crontab to execute the following command every night at 12:00AM, of every day of the month, of every day in the week.
  • “mtime +7” tells the find command to search for files greater than 7 days old

Leave a Reply

Your email address will not be published. Required fields are marked *