Magento 2 update.log increases significant size every minute

Question

UPDATE:

I ended up using logrotate utility to handle the ever-growing log files. I've also set up few cron jobs that will check for unneeded temporary files from time to time.

---

I have the standard Magento 2 cron configured and running. After two days, I encountered some problems while running some actions in the Admin panel. When I checked the server logs, I found out that it was a disk space issue. After further investigation, I found out that this update.log.txt is the culprit. The file size was increasing by>1MB every minute. There was a time it reached more than 20GB.

enter image description here

Is it normal for a standard Magento 2 setup? Or can & how I can disable it or at least minimize it?

It happens to both developer and production mode.

Cron setup was based on this guide.

Cron tab:

* * * * * <path to php binary> <magento install dir>/bin/magento cron:run | grep -v "Ran jobs by schedule" >> <magento install dir>/var/log/magento.cron.log
* * * * * <path to php binary> <magento install dir>/update/cron.php >> <magento install dir>/var/log/update.cron.log
* * * * * <path to php binary> <magento install dir>/bin/magento setup:cron:run >> <magento install dir>/var/log/setup.cron.log

SAMPLE CONTENTS OF THE LOG

/var/www/html/setup/src/Magento/Setup/Console/Command/I18nCollectPhrasesCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/MaintenanceStatusCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/BackupCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/DependenciesShowFrameworkCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/UninstallCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/InfoTimezoneListCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/MaintenanceAllowIpsCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/ModuleStatusCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/InstallStoreConfigurationCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/InfoBackupsListCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/MaintenanceDisableCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/DiCompileCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/InstallCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/UpgradeCommand.php /var/www/html/setup/src/Magento/Setup/Fixtures/IndexersStatesApplyFixture.php /var/www/html/setup/src/Magento/Setup/Fixtures/StoresFixture.php /var/www/html/setup/src/Magento/Setup/Fixtures/CategoriesFixture.php /var/www/html/setup/src/Magento/Setup/Fixtures/SimpleProductsFixture.php /var/www/html/setup/src/Magento/Setup/Fixtures/tax_rates.csv /var/www/html/setup/src/Magento/Setup/Validator /var/www/html/setup/src/Magento/Setup/Validator/DbValidator.php /var/www/html/setup/src/Magento/Setup/Validator/IpValidator.php /var/www/html/setup/src/Magento/Setup/Validator/AdminCredentialsValidator.php /var/www/html/setup/index.php /var/www/html/setup/performance-toolkit /var/www/html/setup/performance-toolkit/.htaccess /var/www/html/setup/performance-toolkit/profiles /var/www/html/setup/performance-toolkit/profiles/ce /var/www/html/setup/performance-toolkit/profiles/ce/large.xml /var/www/html/setup/performance-toolkit/profiles/ce/small.xml /var/www/html/setup/performance-toolkit/profiles/ce/medium.xml /var/www/html/setup/performance-toolkit/profiles/ce/extra_large.xml /var/www/html/setup/performance-toolkit/README.txt /var/www/html/setup/performance-toolkit/benchmark.jmx /var/www/html/vendor/.htaccess [] []

I found few places where this log might be generated / updated :

<INSTALL_DIR>setup\src\Magento\Setup\Console\Command\CronRunCommand.php:
 75 protected function execute(InputInterface $input, OutputInterface $output)
 76 {
 77: $notification = 'setup-cron: Please check var/log/update.log for execution summary.';
 78 
 79 if (!$this->deploymentConfig->isAvailable()) {
 <INSTALL_DIR>setup\src\Magento\Setup\Model\Cron\ReadinessCheck.php:
 164 $write->writeFile(self::SETUP_CRON_JOB_STATUS_FILE, $resultJson);
 165 
 166: // write to permanent log file, var/log/update.log
 167 foreach ($errorLogMessages as $errorLog) {
 168 $this->status->add($errorLog, \Psr\Log\LogLevel::ERROR, false);
 <INSTALL_DIR>setup\src\Magento\Setup\Model\Cron\SetupLoggerFactory.php:
 22 {
 23 $logger = new \Monolog\Logger($channelName);
 24: $path = BP . '/var/log/update.log';
 25 $logger->pushHandler(new \Monolog\Handler\StreamHandler($path));
 26 return $logger;
 <INSTALL_DIR>setup\src\Magento\Setup\Model\Cron\SetupStreamHandler.php:
 17 * @var string
 18 */
 19: protected $fileName = '/var/log/update.log';
 20 
 21 /**
 <INSTALL_DIR>setup\src\Magento\Setup\Model\Cron\Status.php:
 81 $this->varReaderWriter = $filesystem->getDirectoryWrite(DirectoryList::VAR_DIR);
 82 $this->statusFilePath = $statusFilePath ? $statusFilePath : '.update_status.txt';
 83: $this->logFilePath = $logFilePath ? $logFilePath : DirectoryList::LOG . '/update.log';
 84 $this->updateInProgressFlagFilePath = $updateInProgressFlagFilePath
 85 ? $updateInProgressFlagFilePath
 <INSTALL_DIR>setup\src\Magento\Setup\Test\Unit\Model\Cron\StatusTest.php:
 71 $this->varReaderWriter->expects($this->any())
 72 ->method('getAbsolutePath')
 73: ->with('log/update.log')
 74: ->willReturn('DIR/var/log/update.log');
 75: $this->assertEquals('DIR/var/log/update.log', $this->status->getLogFilePath());
 76 }
 77 
 <INSTALL_DIR>update\app\code\Magento\Update\CronReadinessCheck.php:
 16 const CRON_JOB_STATUS_FILE = '.update_cronjob_status';
 17 
 18: const UPDATE_CRON_LOG_FILE = 'var/log/update.log';
 19 
 20 /**#@+
 ..
 91 file_put_contents(MAGENTO_BP . '/var/' . self::CRON_JOB_STATUS_FILE, $resultJson);
 92 
 93: // If non-accessible paths are found, log an 'error' entry for the same in update.log
 94 if ( !$success && !empty($outputString) ) {
 95 $updateLoggerFactory = new UpdateLoggerFactory();
 <INSTALL_DIR>update\app\code\Magento\Update\Status.php:
 67 ) {
 68 $this->statusFilePath = $statusFilePath ? $statusFilePath : MAGENTO_BP . '/var/.update_status.txt';
 69: $this->logFilePath = $logFilePath ? $logFilePath : MAGENTO_BP . '/var/log/update.log';
 70 $this->updateInProgressFlagFilePath = $updateInProgressFlagFilePath
 71 ? $updateInProgressFlagFilePath
 <INSTALL_DIR>update\app\code\Magento\Update\UpdateLoggerFactory.php:
 29 */
 30 public function __construct(
 31: $logFile = '/var/log/update.log',
 32 $channelName = 'update-cron'
 33 ) {
 <INSTALL_DIR>update\cron.php:
 15 $status = new \Magento\Update\Status();
 16 $cronReadinessChecker = new \Magento\Update\CronReadinessCheck();
 17: $notification = 'update-cron: Please check var/log/update.log for execution summary.' . PHP_EOL;
 18 
 19 if (!$cronReadinessChecker->runReadinessCheck()) {
 <INSTALL_DIR>update\dev\tests\integration\testsuite\Magento\Update\StatusTest.php:
 43 $this->statusFilePath = __DIR__ . '/_files/update_status.txt';
 44 $this->tmpStatusFilePath = TESTS_TEMP_DIR . '/update_status.txt';
 45: $this->tmpStatusLogFilePath = TESTS_TEMP_DIR . '/update.log';
 46 $this->updateInProgressFlagFilePath = TESTS_TEMP_DIR . '/update_in_progress.flag';
 47 $this->updateErrorFlagFilePath = TESTS_TEMP_DIR . '/update_error.flag';

---

UPDATE:

I ended up using logrotate utility to handle the ever-growing log files. I've also set up few cron jobs that will check for unneeded temporary files from time to time.

• Budget Woodworker: I'm having the same issue with my store. My update.log file was 23g and actually crashed my mysql last night. I'm not sure what's going on - i have a fairly vanilla install.

Answer 1

You probably set wrong permissions on some folders. /update/cron lists all folders with wrong permissions

Easy way is just comment out in crontab update and setup

#* * * * * <path to php binary> <magento install dir>/update/cron.php >> <magento install dir>/var/log/update.cron.log
#* * * * * <path to php binary> <magento install dir>/bin/magento setup:cron:run >> <magento install dir>/var/log/setup.cron.log

You don't need them if you don't need to update and install from admin panel.

It's dangerous idea for production server. You can manually update via composer on test/local server; test it; then copy updates to production server.

The one you really need is

* * * * * <path to php binary> <magento install dir>/bin/magento cron:run | grep -v "Ran jobs by schedule" >> <magento install dir>/var/log/magento.cron.log

It manages "indexers, sends automated e-mails, generates the sitemap, and so on" - as described in devdocs

• reuelab: Yes, this was also indicated in the Magento 2 documentation for configuring cron. Anyway we ended up using logrotate and some custom cron jobs, that check and clean the file system, with the goal to optimize the disk space usage.

Answer 2

As far as I know, the var/log/update.log file is used with the cron indeed.

It is declared in \setup\src\Magento\Setup\Model\Cron\Status.php

$this->logFilePath = $logFilePath ? $logFilePath : DirectoryList::LOG . '/update.log';

And then retrieved via the getLogFilePath() method.

If you check where this method is being called, it's under the \setup\src\Magento\Setup\Model\Cron\JobFactory.php :

$logStream = fopen($cronStatus->getLogFilePath(), 'a+');
$streamOutput = new MultipleStreamOutput([$statusStream, $logStream]);

Then the $streamOutput variable is passed as a parameter to the job class, for example:

return new JobUpgrade(
 $this->serviceLocator->get('Magento\Setup\Console\Command\UpgradeCommand'),
 $objectManagerProvider,
 $streamOutput,
 $this->serviceLocator->get('Magento\Setup\Model\Cron\Queue'),
 $cronStatus,
 $name,
 $params
);

If I keep going with this example, the $this->output is used in the execute() method:

$this->command->run(new ArrayInput($this->params), $this->output);

The run() method is declared in vendor\symfony\console\Symfony\Component\Console\Command\Command.php

The $output variable is then used in several places such as :

 $this->initialize($input, $output);

As well as :

 if ($input->isInteractive()) {
 $this->interact($input, $output);
 }
 $input->validate();
 if ($this->code) {
 $statusCode = call_user_func($this->code, $input, $output);
 } else {
 $statusCode = $this->execute($input, $output);
 }

Another interesting one:

 } elseif (OutputInterface::VERBOSITY_VERY_VERBOSE === $output->getVerbosity()) {
 $output->writeln('<comment>Install the proctitle PECL to be able to change the process title.</comment>');
 }

According to this last code, some stuff can be written in case of very verbose parameter. Maybe you setup your cron jobs with -vv and that's why stuff is getting logged.

This is not a definite answer, but you should follow that stack trace I just gave you to investigate the issue.

• akio an: Thank you! this is only the answer which I wanted.

Answer 3

/var/log/update.log is cron log file and this issue especially comes when you have files permission issue. "This is what i got in my case when i open that file."

and this could be the fix: https://community.bitnami.com/t/update-log-file-grows-rapidly-in-magento-2-1-1/49253/6

You need to set correct permissions and ownership to your Magento directories and files.

Answer 4

As per I know update.log is your custom log file because Magento 2 generate log this kind of files

• var/log/debug.log – all information that is saved by Psr\Log\LoggerInterface::debug() is stored there.
• var/log/exception.log – all exceptions information is stored there.
• var/log/system.log – information from other logging levels is saved there.

I am not sure why It generate so much log data in your file might be some syntax error or something like that.

But I would suggest just search with the update.log with in your Magento 2 code directory and remove that log from your code.

• reuelab: Hi, Thanks for answering. I dont think its a custom log, cause i can find it on some default Magento Classes. I've updated my question for more information.