External variable in callback for chunk()

Question

I am trying to implement search in Laravel. I want to search on models. Since there can be many records in the database, I am trying the chunk function.

public function searchLeads($param)
 {
 $results = array();
 // get all leads
 Lead::chunk(100, function($leads) use ($results, $param) {
 // search in name
 $results = array_merge($results, $this->repository->searchInName($leads, $param));
 // search in email
 $results = array_merge($results, $this->repository->searchInEmail($leads, $param));
 // and so on ...
 }
 // eliminate duplicates
 $collection = collect($results);
 $collection = $collection->unique();
 $results = $collection->all();
 dd($results);
 // return $results;
}

Note: The searchIn... functions return array of results and are working as expected.

When I try the above code, I get an empty array. So I changed the following line (added a reference & to $results).

Lead::chunk(100, function($leads) use (&$results, $param) {

Now I get the expected output.

Am I doing this the correct way? Have I stumbled upon the right solution, or am I missing something that might introduce bugs in the code?

Answer 1

I got the solution after reading a bit about php closures. My real worry was that I am not using the $results array correctly.

But after reading php docs, I know I am doing it correctly.

Please note that I am aware of where clause and it's usage would be much more efficient here. But that's not what I was trying to ask.

Answer 2

You managed to bypass the query builder, which could have implemented this as a trivial, single query on the backing database, and instead managed to emulate that query in an very inefficient manner yourself.

// get all leads
Lead::chunk(100, function($leads) use ($results, $param) {
 // search in name
 $results = array_merge($results, $this->repository->searchInName($leads, $param));
 // search in email
 $results = array_merge($results, $this->repository->searchInEmail($leads, $param));
 // and so on ...
}

This effectively causes 2 additional queries to the database for each 100 results in the Lead model. This might still work reasonably in your test environment where you have at most a few dozen leads registered, but will crawl to a halt in production as the number of leads increases with time.

Even worse, it requires you to stream all existing Lead instances through PHP, even if they are not even part of the final result set.

• linuxartisan: So what do you suggest? If you are suggesting using where clause, let me tell you that's not possible. I am not strictly searching on database columns. There are derived values and related records that need to be searched; hence those searchIn...functions.
• Ext3h: @linuxartisan Searching in related records and derived values is what your relational database is good at. Hence I doubt that you couldn't state your query in a compact form using the query builder. Maybe you should add the source for searchInName and searchInEmail, as well as the Lead model to your original question, as they appear to be part of the problem.

Answer 3

The chunk call should replace the get call - it is designed to work on the pre-get() query builder object.

means you need to change Lead::chunk to
$leads = Lead::orderBy('created','desc');
$leads->chunk(100, function($leads) use ($results, $param) {
 // search in name
 $results = array_merge($results, $this->repository->searchInName($leads, $param));
 // search in email
 $results = array_merge($results, $this->repository->searchInEmail($leads, $param));
 // and so on ...
 } 

and for more do not need of method $collection->all()

$collection = collect($results);
$collection = $collection->unique(); 
$return = $collection->all(); 
dd($return);

• Kuldeep Bhagvandas Chhatala: please check the edited answer, reference stackoverflow.com/a/38551313/1363245
• linuxartisan: $collection->all() is for converting the collection to array.