Memory leak in magento when saving products

Question

I have a problem when trying to mass update group prices in products. It seems when saving the products memory doesn't get released properly and keeps accumulating until it gets exhausted. I've set the limit to 4G as well and behavior was consistent.

Using memory_get_usage I can track the issue down to the product->save() function which doesn't seem to release memory correctly.

 <?php 
 require_once(__DIR__ . '/../app/Mage.php');
 $app = Mage::app();
//choose admin view
 $app->getStore()->setId(Mage_Core_Model_App::ADMIN_STORE_ID);
 $priceData = [...]
 $loop = 1;
 $model = Mage::getModel('catalog/product');
 foreach ($priceData as $priceArray) {
 if (!empty($priceArray[0])) {
 $product = Mage::getModel('catalog/product')->loadByAttribute('sku', $priceArray[0]);
 if (isset($product)) {
 $product->load($product->getId());
 Mage::log('processing product: ' . $product->getId());
 $group_prices = $product->getData('group_price');
 if (is_null($group_prices)) {
 $attribute = $product->getResource()->getAttribute('group_price');
 if ($attribute) {
 $attribute->getBackend()->afterLoad($product);
 $group_prices = $product->getData('group_price');
 }
 }
 if(!is_array($group_prices)){
 $group_prices = array();
 }
 if (!empty($priceArray[1])) {
 $priceGroupA = array(array(
 'website_id' => 0,
 'cust_group' => 6,
 'price' => (float) $priceArray[1]
 ))
 ;
 $group_prices = array_merge($group_prices, $priceGroupA);
 }
 if (!empty($priceArray[2])) {
 $priceGroupB = array(array(
 'website_id' => 0,
 'cust_group' => 5,
 'price' => (float) $priceArray[2]
 ))
 ;
 $group_prices = array_merge($group_prices, $priceGroupB);
 }
 $retailPrice = max([(float)$priceArray[1], (float)$priceArray[2], 0]);
 $product->setData('price', $retailPrice);
 $product->setData('group_price', $group_prices);
 Mage::log(memory_get_usage(true)); 
 $product->save();
 Mage::log(memory_get_usage(true))
 $loop++;
 }
 }
 }

I can see that after a loop run has finished the memory is increased by 50MB approximately and then just before product->save() memory usage is decreased slightly and then it increases by about 50MB again.

I've tried calling

$product->clearInstance();
$product=null;
unset($product);
$product->unsetData();

I'm using Magento 1.9.3.6 and all previous relevant answers I found appear to be for older versions and don't seem to be valid any more.

Example log

2017年09月24日T06:58:26+00:00 DEBUG (7): processing product: 2032
2017年09月24日T06:58:26+00:00 DEBUG (7): 35127296
2017年09月24日T06:58:36+00:00 DEBUG (7): 99614720
2017年09月24日T06:58:36+00:00 DEBUG (7): loop 345 finished
2017年09月24日T06:58:37+00:00 DEBUG (7): processing product: 2033
2017年09月24日T06:58:37+00:00 DEBUG (7): 99614720
2017年09月24日T06:58:47+00:00 DEBUG (7): 146538496
2017年09月24日T06:58:47+00:00 DEBUG (7): loop 346 finished
2017年09月24日T06:58:47+00:00 DEBUG (7): processing product: 2034
2017年09月24日T06:58:47+00:00 DEBUG (7): 146538496
2017年09月24日T06:58:57+00:00 DEBUG (7): 188481536
2017年09月24日T06:58:57+00:00 DEBUG (7): loop 347 finished
2017年09月24日T06:58:58+00:00 DEBUG (7): processing product: 2035
2017年09月24日T06:58:58+00:00 DEBUG (7): 188481536
2017年09月24日T06:59:09+00:00 DEBUG (7): 230162432
2017年09月24日T06:59:09+00:00 DEBUG (7): loop 348 finished
2017年09月24日T06:59:09+00:00 DEBUG (7): processing product: 2036
2017年09月24日T06:59:09+00:00 DEBUG (7): 229900288

Any idea is appreciated because the only good workaround that I could find is is using ajax and doing them in small batches but that's not something that would work consistently accross multiple hosting environments without tweaking. I'd rather see if there's a way to solve the leak.

Magento customer save memory leak

• Rick Buczynski: I won't submit an answer because I simply don't know the cause. However, I wanted to comment on your sample code given. Save some overhead by using $product->getIdBySku() to acquire the ID before loading the product model. Also, for standard attributes you can avoid full product save calls by instead calling $product->getResource()->saveAttribute($product, 'attribute_code') for each attribute.
• gabtzi: Hi Rick, for group prices saveAttribute won't work I'm afraid so saving the full product was my only option there. Thanks for the pointers though, much appreciated
• Rick Buczynski: Good point. I should have checked into that but was responding from my phone! I am certain the save routine for group pricing is managed in the product resource model, so if you study Mage_Catalog_Model_Resource_Product you might be able to optimize further.
• gabtzi: It's OK :) Last time I looked up how to save only group_price I concluded that it would require a rewrite of the model to actually perform it so I opted against it. It's not something that will happen very often so I could live with the speed overhead since it's a limited product set. But the memory leak is a showstopper. I noticed that when disabling the saveBefore and saveAfter callbacks in Mage_Core_Model_Abstract it's then lightning fast and memory much lower so it's probably maintaining a reference to the Varien_Object there and it can't be removed by the garbage collector.

Answer 1

I took a look into this out of curiosity, and made some interesting observations, though I don't think I can fully answer your question at its core. First, the observations:

• Most expensive method is Mage_Catalog_Model_Product::afterCommitCallback
  • Indexers run here on each product save
• Next most expensive are Mage_Catalog_Model_Product::_[before|after]Save
  • For this reason most articles will advocate individual attribute saves

Even after turning some of these core-level routines off while keeping your existing sample code together, I was seeing ~20% decrease in memory consumption; however, usage continued to increase into 40MB by the end of a run on ~300 test SKUs.

At this point I was more interested in the "tried and true" optimal save routines that Magento would recommend in this scenario: bypassing the front model to leverage the resource model.

Here is my modified example of your code. Because I did not have price data to work with, you will see a block of my code where I am generating it for use:

require_once('/vagrant/public_html/app/Mage.php');
ini_set('display_errors', 1);
Mage::app()->setCurrentStore(Mage_Core_Model_App::ADMIN_STORE_ID);
$priceData = array();
// START: Mock data for testing
$resource = Mage::getResourceModel('catalog/product');
$adapter = $resource->getReadConnection();
$select = $adapter->select()
 ->from($resource->getEntityTable(), array('sku'));
$skus = $adapter->fetchCol($select);
foreach ($skus as $sku) {
 $priceData[] = array(
 $sku,
 mt_rand(100,500),
 mt_rand(10,75),
 );
}
$cGroupA = 3;
$cGroupB = 2;
$total = count($priceData);
// END: Mock data for testing
$loop = 1;
$product = Mage::getModel('catalog/product');
$resource = $product->getResource();
$lastLength = null;
echo 'Processing products...' . PHP_EOL;
echo 'Starting usage: ' . round(memory_get_usage(true) / 1000000, 1) . 'M' . PHP_EOL;
foreach ($priceData as $priceArray) {
 if (!empty($priceArray[0])) {
 $product->reset()->load($product->getIdBySku($priceArray[0]));
 if ($product->getId()) {
 $groupPriceData = (array) $product->getData('group_price');
 $groupPriceAttr = $product->getResource()->getAttribute('group_price');
 if (is_null($groupPriceData) && $groupPriceAttr) {
 $groupPriceAttr->getBackend()->afterLoad($product);
 $groupPriceData = $product->getData('group_price');
 }
 if (!empty($priceArray[1])) {
 $priceGroupA = array(array(
 'website_id' => 0,
 'cust_group' => $cGroupA,
 'price' => (float) $priceArray[1]
 ));
 $groupPriceData = array_merge($groupPriceData, $priceGroupA);
 }
 if (!empty($priceArray[2])) {
 $priceGroupB = array(array(
 'website_id' => 0,
 'cust_group' => $cGroupB,
 'price' => (float) $priceArray[2]
 ));
 $groupPriceData = array_merge($groupPriceData, $priceGroupB);
 }
 $retailPrice = max(array((float)$priceArray[1], (float)$priceArray[2], 0));
 $product->setData('price', $retailPrice);
 $product->setData('group_price', $groupPriceData);
 $resource->saveAttribute($product, 'price');
 $groupPriceAttr->getBackend()->afterSave($product);
 if ($lastLength) {
 echo "033円[{$lastLength}D";
 }
 $usageString = "> " . round(memory_get_usage(true) / 1000000, 1) . 'M (' . round($loop/$total, 2) * 100 . '%)';
 $lastLength = strlen($usageString);
 echo $usageString;
 $loop++;
 }
 }
}

Key takeaways:

• Note my generated price data, please use your real data
• Note that I modified your script to output a progress bar over CLI
• I am leveraging the resource models to do the save work
• I was able to keep memory "leaks" down to about +/- 5MB this way

Hope this helps to solve your issue.

• gabtzi: This was the perfect answer on how to save only the group prices really and very interesting. I was able to go through the whole lot of 400 SKUs. Memory remained at 183.5M while steadily increasing as you mentioned by a few megabytes each loop. However I hope you don't mind me not marking it yet as an answer since the underlying issue is just transferred somewhere where it's not as expensive and it would still exist in larger datasets. I really do appreciate your effort though and it really helped me at least use it with relative safety .
• Rick Buczynski: You're welcome, and I understand. I'm still wondering for myself how to address the root. I am almost certain it can't be resolved without a series of core rewrites, given how many object references are established during a save call. Then there is also the issue of accounting for event observers. If I get any closer I'll answer again.