Avoiding Google Maps geocode limit?

Question

I'm creating a custom google map that has 125 markers plotted via a cms. When loading the map I get this message:

Geocode was not successful for the following reason: OVER_QUERY_LIMIT

I'm pretty sure it's the way in which I've geocoded the markers.

How can I avoid these warnings and is there a more efficient way to geocode the results?

UPDATE: This is my attempt at Casey's answer, I'm just getting a blank page at the moment.

<script type="text/javascript"> 
(function() { 
window.onload = function() { 
 var mc;
// Creating an object literal containing the properties we want to pass to the map 
var options = { 
zoom: 10, 
center: new google.maps.LatLng(52.40, -3.61), 
mapTypeId: google.maps.MapTypeId.ROADMAP 
}; 
// Creating the map 
var map = new google.maps.Map(document.getElementById('map'), options); 
// Creating a LatLngBounds object 
var bounds = new google.maps.LatLngBounds(); 
// Creating an array that will contain the addresses 
var places = []; 
// Creating a variable that will hold the InfoWindow object 
var infowindow; 
mc = new MarkerClusterer(map);
<?php
$pages = get_pages(array('child_of' => $post->ID, 'sort_column' => 'menu_order'));
$popup_content = array();
foreach($pages as $post)
 {
 setup_postdata($post);
 $fields = get_fields(); 
 $popup_content[] = '<p>'.$fields->company_name.'</p><img src="'.$fields->company_logo.'" /><br /><br /><a href="'.get_page_link($post->ID).'">View profile</a>';
 $comma = ", ";
 $full_address = "{$fields->address_line_1}{$comma}{$fields->address_line_2}{$comma}{$fields->address_line_3}{$comma}{$fields->post_code}";
 $address[] = $full_address;
 }
wp_reset_query();
echo 'var popup_content = ' . json_encode($popup_content) . ';';
echo 'var address = ' . json_encode($address) . ';';
?>
var geocoder = new google.maps.Geocoder(); 
var markers = [];
// Adding a LatLng object for each city 
for (var i = 0; i < address.length; i++) { 
 (function(i) { 
 geocoder.geocode( {'address': address[i]}, function(results, status) {
 if (status == google.maps.GeocoderStatus.OK) {
 places[i] = results[0].geometry.location;
 // Adding the markers 
 var marker = new google.maps.Marker({position: places[i], map: map});
 markers.push(marker);
 mc.addMarker(marker);
 // Creating the event listener. It now has access to the values of i and marker as they were during its creation
 google.maps.event.addListener(marker, 'click', function() {
 // Check to see if we already have an InfoWindow
 if (!infowindow) {
 infowindow = new google.maps.InfoWindow();
 }
 // Setting the content of the InfoWindow
 infowindow.setContent(popup_content[i]);
 // Tying the InfoWindow to the marker 
 infowindow.open(map, marker);
 });
 // Extending the bounds object with each LatLng 
 bounds.extend(places[i]); 
 // Adjusting the map to new bounding box 
 map.fitBounds(bounds) 
 } else { 
 alert("Geocode was not successful for the following reason: " + status); 
 }
 });
 })(i);
} 
var markerCluster = new MarkerClusterer(map, markers); 
} 
})
(); 
</script> 

It doesn't really matter what the solution is as long as the markers load instantly and it's not breaking any terms & conditions.

• Peter: You dont say why or how often you are doing these lookups. You arent trying to geocode the same points every page render are you? If thats the case you should be looking at using the api to do these lookups server side once only, when the location is first added to the database.
• Stephen Lead: If you want to try @Peter's advice, there's a simple tool here which will geocode the points in a Google spreadsheet. This would allow you write the lat/long for the cities into the CMS, so you wouldn't need to use the geocoder at run-time.
• Rob: @Peter Actually that sounds like a great idea. I'm a little torn as it will decrease the usability of my cms. If the client is having to find out lat/long per company rather than just entering the address then it's not very user friendly. Am I understanding that correctly?
• Peter: See my new answer lower down for a php function.
• Stephen Lead: @rob FWIW I think Ragi has the right approach. Casey's answer still requires that you geocode every point, every time the map draws, which doesn't appear to be necessary for your static dataset

Answer 1

Like everybody else, I could give you an answer with code, but I don't think somebody has explained to you that you are doing something that is fundamentally wrong.

Why are you hitting this error? Because you are calling geocode every time somebody views your page and you are not caching your results anywhere in the db!

The reason that limit exists is to prevent abuse from Google's resources (whether it is willingly or unwillingly) - which is exactly what you are doing :)

Although google's geocode is fast, if everybody used it like this, it would take their servers down. The reason why Google Fusion Tables exist is to do a lot of the heavy server side lifting for you. The geocoding and tile caching is done on their servers. If you do not want to use that, then you should cache them on your server.

If still, 2500 request a day is too little, then you have to look at Google Maps Premier (paid) license that gives you 100,000 geocoding requests per day for something around 10k a year (that is a lot - with server side caching you should not be reaching this limit unless you are some huge site or are doing heavy data processing). Without server side caching and using your current approach, you would only be able to do 800 pageviews a day!

Once you realize that other providers charge per geocode, you'll understand that you should cache the results in the db. With this approach it would cost you about 10 US cents per page view!

Your question is, can you work around the throttle limit that Google gives you? Sure. Just make a request from different ip addresses. Heck, you could proxy the calls through amazon elastic ips and would always have a new fresh 2500 allotted calls. But of course, besides being illegal (you are effectively circumventing the restriction given to you by the Google Maps terms of service), you would be doing a hack to cover the inherent design flaw you have in your system.

So what is the right way for that use-case? Before you call the google geocode api, send it to your server and query if it is in your cache. If it is not, call the geocode, store it in your cache and return the result.

There are other approaches, but this should get you started in the right direction.

Update: From your comments below, it said you are using PHP, so here is a code sample on how to do it correctly (recommendation from the Google team itself) https://developers.google.com/maps/articles/phpsqlsearch_v3

• Rob: Have you got an example I could follow of sending it to the server etc?
• Ragi Yaser Burhum: What is your backend CMS? (I assume you can/know how change it and just want a simple example that shows how to do that in your language of choice. You don't need an example of a client side js, or do you? Are you using any js framework like jquery?
• Ragi Yaser Burhum: Also, you will need to give more context about how the marker data gets into your system in the first place
• Ragi Yaser Burhum: OK. Php then. This bit from the Google Documentation should include the mysql and php part code.google.com/apis/maps/articles/phpsqlgeocode.html
• Paul Ramsey: See 10.1.3 (b) for details on what kinds of caching are allowed. developers.google.com/maps/terms#section_10_12

Answer 2

I think Sasa is right, on both counts.

In terms of sending all your addresses at once, one option is to send the requests at intervals. In the past when using JavaScript I have opted to delay requests by 0.25 (seems to work!) seconds using the

setTimeout( [FUNCTION CALL] , 250 )

method.

In .NET i have opted for:

System.Threading.Thread.Sleep(250);

Seems to work.

EDIT: Cant really test it, but this should/might work!

Javascript example. The addressArray holds strings that are addresses...

for (var i = 0; i < addressArray.length; i++0 
{
setTimeout('googleGeocodingFunction(' + addressArray[i] + ')' , 250);
}

EDIT:

for (var i = 0; i < address.length; i++) {
 function(i) {
 setTimeout(geocoder.geocode({ 'address': address[i] }, function(results, status) {
 if (status == google.maps.GeocoderStatus.OK) {
 places[i] = results[0].geometry.location;
 var marker = new google.maps.Marker({ position: places[i], map: map });
 markers.push(marker);
 mc.addMarker(marker);
 google.maps.event.addListener(marker, 'click', function() {
 if (!infowindow) {
 infowindow = new google.maps.InfoWindow();
 }
 // Setting the content of the InfoWindow
 infowindow.setContent(popup_content[i]);
 // Tying the InfoWindow to the marker 
 infowindow.open(map, marker);
 });
 // Extending the bounds object with all coordinates 
 bounds.extend(places[i]);
 // Adjusting the map to new bounding box 
 map.fitBounds(bounds)
 } else {
 alert("Geocode was not successful for the following reason: " + status);
 }
 }), 250);// End of setTimeOut Function - 250 being a quarter of a second. 
 } 
}

• Rob: Thanks for the reply. How can I get that into my code. This is all a bit new so need some hand holding. You should be able to view my source code.
• CatchingMonkey: Which bit? What language are you using?
• Rob: I'm using javascript, how could I set an interval?
• CatchingMonkey: Edited my answer.
• CatchingMonkey: Also realsed i meant setTimeout()... Thanks Casey.

Answer 3

It sounds like you are hitting the simultaneous request limit imposed by Google (though I cannot find a reference to what the limit actually is). You will need to space your requests out so that you do not send 125 requests all at once. Note that there is also a 2500 geocode request per day limit.

Consult the Google Geocoding Strategies document for more information.

Update: As an added solution inspired thanks to a post by Mapperz, you could think about creating a new Google Fusion Table, storing your address and any related data in the table, and geocoding the table through their web interface. There is a limit to the number of geocode requests a Fusion Table will make, however the limit is quite large. Once you reach the limit, you will need to re-submit the geocode request, and Fusion Tables will pick up where it left off. Bonus to this approach is a huge speed improvement, as you will only need to geocode ONCE, where with your current solution you will need to geocode on every load, quickly reaching daily limits.

• Rob: I guessed I'd be hitting the limit, I'm so new to this that I think I need a code example to help.
• Rob: I've been looking into Google Fusion Tables and come across a few problems. 1. You have to manually geocode addresses (via file->geocode) in the table, can this be done automatically? 2. Some markers are only appearing at certain zoom levels while others show whatever the zoom level. This is the example - mediwales.com/mapping/example. What I eventually want to do is export my Wordpress data (ie all the company addresses) to the fusion table, geocode those automatically then grab all those geocoded addresses and plot the map instantly.

Answer 4

Here is how I throttle my geocode requests in javascript, addresses being an array containing each address to geocode:

for (i=0;i<=addresses.length;i++) {
 setTimeout( function () {
 geocoder.geocode( { 'address': addresses[i]}, function(results, status) {
 if (status == google.maps.GeocoderStatus.OK) { 
 //create and add marker to map based off geocode result
 var marker = new google.maps.Marker({ 
 map: map,
 title: results[0].formatted_address,
 position: results[0].geometry.location
 });
 } //EDIT, was missing the closing bracket here
 });
 }, i * 1500);
}

This essentially adds a second and half between each geocoding request.

• Rob: Thanks for the answer. I keep getting a blank page when trying to integrate into my code - mediwales/mapping
• Casey: Are you using any debugging tools that could point out the error? If you aren't, try using the FireFox plugin firebug. It can be pretty handy in situations like this.
• Rob: I'm not no. I'm not confident I've put the code in the right place, I'll update my question with my attempt.
• Casey: I was missing a closing bracket in my code sample. I edited my answer and added it, with a comment. However, it looks like this wasn't an issue in your sample code. Are you getting a blank page or a blank map? A blank page would indicate there is a syntax error in the javascript code. A blank map (no markers) would indicate that the geocoding code has an issue.
• Rob: Hi. I've had another go at your answer. The delay is working now but can't get the markers to plot. From my code these are the important bits that plot the markers - (function(i) { and })(i); without those it won't plot, even without adding the delay.

Answer 5

I have tried the page now and it seems to be working right now.

While I think the the timer approach you are currently using is correct and as far as I know the only one that will work w/o requiring some server side support I should suggest the following changes (I took the liberty of rewriting your geocoding functions) to take advantage of localStorage.

LocalStorage is a relatively new technology built into most browsers that allows the web page to store data on the client. It somewhat similar to cookies, but it is much more powerful and it is not always resent to the server (unlike cookies).

My aim is to save geocoded address on the client so that if a user refreshes the page there is no need to invoke Google's geocoder functions again on them. The implementation below is crude (we should add code to check for stale records and probably need a better caching key than the index) but should be enough to get you started. Hope it helps.

(function() { 
window.onload = function() { 
 var mc;
// Creating an object literal containing the properties we want to pass to the map 
var options = { 
zoom: 0, maxZoom: 0, 
center: new google.maps.LatLng(52.40, -3.61), 
mapTypeId: google.maps.MapTypeId.ROADMAP 
}; 
// Creating the map 
var map = new google.maps.Map(document.getElementById('map'), options); 
// Creating a LatLngBounds object 
var bounds = new google.maps.LatLngBounds(); 
// Creating an array that will contain the addresses 
var places = []; 
// Creating a variable that will hold the InfoWindow object 
var infowindow; 
mc = new MarkerClusterer(map);
var popup_content = [..redacted..];
var geocoder = new google.maps.Geocoder(); 
var markers = [];
// Adding a LatLng object for each city 
function geocodeAddress(i) {
 // check if localStorage is available (it is now available on most modern browsers)
 // http://html5tutorial.net/tutorials/working-with-html5-localstorage.html
 // NB: this *must* be improved to handle quota limits, age/freshness, etc
 if(localStorage && localStorage['address_'+i]) {
 places[i]=JSON.parse(localStorage['address_'+i]);
 addPlace(i);
 } else {
 geocoder.geocode( {'address': address[i]}, function(results, status) {
 if (status == google.maps.GeocoderStatus.OK) {
 places[i] = results[0].geometry.location;
 if(localStorage) {
 // cache result locally on the browser, this will help reducing the number of requests
 // to the google geocoder in case the user refreshes the page
 // remember: the more he will refresh, the more he's likely to hit the limit
 // (this is one case where refreshing or closing the browser does not work)
 localStorage['address_'+i]=JSON.stringify(results[0].geometry.location);
 }
 addPlace(i);
 } else { 
 console.log("Geocoding of address "+address[i]+" failed");
 }
 })
}
function addPlace(i) {
 // Adding the markers 
 var marker = new google.maps.Marker({position: places[i], map: map});
 markers.push(marker);
 mc.addMarker(marker);
 // Creating the event listener. It now has access to the values of i and marker as they were during its creation
 google.maps.event.addListener(marker, 'click', function() {
 // Check to see if we already have an InfoWindow
 if (!infowindow) {
 infowindow = new google.maps.InfoWindow();
 }
 // Setting the content of the InfoWindow
 infowindow.setContent(popup_content[i]);
 // Tying the InfoWindow to the marker 
 infowindow.open(map, marker);
 });
 // Extending the bounds object with each LatLng 
 bounds.extend(places[i]); 
 // Adjusting the map to new bounding box 
 map.fitBounds(bounds);
}
function geocode() {
 if (geoIndex < address.length) {
 geocodeAddress(geoIndex);
 ++geoIndex;
 }
 else {
 clearInterval(geoTimer);
 }
}
var geoIndex = 0;
var geoTimer = setInterval(geocode, 600); // 200 milliseconds (to try out)
var markerCluster = new MarkerClusterer(map, markers); 
} 
})
(); 

• Rob: Thanks very much, just giving something else a go but will try this in a minute. Sounds like a good idea.
• Rob: Just tried it out (have now reverted back to previous), it worked the first time then when I refreshed it just showed a blank page.
• unicoletti: try now, there was an error, also please note that the popup_content variable has been redacted for brevity.

Answer 6

using geopy geocoders I'm able to geocode a large number of addresses without a hitch (i'm not sure of the posters setup, or if he can include a python dependency to his project)

here is my test:

from geopy import geocoders
g = geocoders.GoogleV3() #https://geopy.readthedocs.io/en/latest/#geopy.geocoders.GoogleV3
for x in xrange(1,10000):
 print x
 a = g.geocode("AV JOAO NAVES DE AVILA " + str(x) + " UBERLANDIA MG BRASIL")
 print a

result snippet

7595 (u'Av. Jo\xe3o Naves de \xc1vila, 7595 - Segismundo Pereira, Uberl\xe2ndia - Minas Gerais, 38408-288, Brazil', (-18.9388154, -48.2220562))

7596 (u'Av. Jo\xe3o Naves de \xc1vila, 7596 - Segismundo Pereira, Uberl\xe2ndia - Minas Gerais, 38408-680, Brazil', (-18.938814, -48.2217423))

7597 (u'Av. Jo\xe3o Naves de \xc1vila, 7597 - Segismundo Pereira, Uberl\xe2ndia - Minas Gerais, 38408-288, Brazil', (-18.9388128, -48.2220381))

7598 (u'Av. Jo\xe3o Naves de \xc1vila, 7598 - Segismundo Pereira, Uberl\xe2ndia - Minas Gerais, 38408-680, Brazil', (-18.9388114, -48.2217242))

7599 (u'Av. Jo\xe3o Naves de \xc1vila, 7599 - Segismundo Pereira, Uberl\xe2ndia - Minas Gerais, 38408-288, Brazil', (-18.9388102, -48.2220201))

7600 (u'Av. Jo\xe3o Naves de \xc1vila, 7600 - Segismundo Pereira, Uberl\xe2ndia - Minas Gerais, 38408-680, Brazil', (-18.9388088, -48.2217062))

This is still running and I'm with 7k results.

EDIT: After a while i've hitted the end of the street and google started to give me same points, but looks like it's still making requests and giving me accurate answers.

• alphabetasoup: Geopy isn't magic, the usage constraints still apply.

Answer 7

This php function (i dont know which language youre working in, but you can pick it apart im sure) uses google geolocator api. Given an address, it will return the normalised address and latlong. HTH.

// GEOLOCATEBYADDRESS
// @arg (string)address
// @return (array) ((int)flag,(array)address,array(rawresponse))
function geolocatebyaddress($lookupaddress){
 $lookupaddress=trim("$lookupaddress New Zealand");
 $lookupaddress=urlencode($lookupaddress);
 //send off google api lookup request
 $apiurl="http://maps.google.com/maps/api/geocode/json";
 $apiurl.="?address=$lookupaddress";
 $apiurl.="&region=NZ";
 $apiurl.="&sensor=false";
 if (function_exists("curl_init")) {
 $ch = curl_init();
 curl_setopt($ch, CURLOPT_URL, $apiurl);
 curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
 $json = curl_exec($ch);
 curl_close($ch);
 }
 else $json= file_get_contents($apiurl);
 //process response
 $response= json_decode($json,true);
 $approxflag=0;
 if ($response['status']=='OK' and count($response['results'])) {
 $aa= array();
 foreach($response['results'] as $cc=>$result) {
 //set keys
 foreach(array('street_number','road','suburb','city','postcode','region','lat','long') as $t){
 $aa[$cc][$t]='';
 }
 if ($result['geometry']){
 $aa[$cc]['lat']= $result['geometry']['location']['lat'];
 $aa[$cc]['long']=$result['geometry']['location']['lng'];
 }
 if (count($result['address_components'])){
 foreach ($result['address_components'] as $acs){
 if ($acs['types'][0]=='street_number') $aa[$cc]['street_number']= $acs['long_name']; 
 if ($acs['types'][0]=='route') $aa[$cc]['road']= $acs['long_name']; 
 if ($acs['types'][0]=='sublocality') $aa[$cc]['suburb']= $acs['long_name']; 
 if ($acs['types'][0]=='locality') $aa[$cc]['city']= $acs['long_name']; 
 if ($acs['types'][0]=='postal_code') $aa[$cc]['postcode']= $acs['long_name']; 
 if ($acs['types'][0]=='administrative_area_level_1') $aa[$cc]['region']= $acs['long_name']; 
 } 
 }
 //successful?
 if ($result['geometry']['location_type']=='APPROXIMATE') $approxflag++;
 if (isset($result['partial_match'])) $approxflag++;
 if (!$aa[$cc]['street_number']) $approxflag++;
 }
 if ($approxflag) return (array(1,$aa,$response));
 else return (array(2,$aa,$response));
 }
 else return (array(0,array(),$response));
} 

• Rob: Thanks for the reply, how would I incorporate this into my code?
• Peter: Sorry i presumed you were coding the site. That function is used as part of server side scripting that for instance takes a user input address, uses the google geocoder api to get the coords. You can then put those coords in your db, and when you need to produce a map, just query the coords and produce the map with markers. That way you wont be abusing the geocoder for every page render as dicsussed. (did you downote my comment? I am trying to help, and penalising me isnt going too encourage me at all). Also i forogt to mention that the api requires you to use it with a gmap.

Answer 8

Just now saw your query. You can try following code. On the geocode status check else part, recursively call the same function with a time gap for the lost addresses.

function spotPosition(address) {
 geocoder.geocode({ 'address': address }, function (results, status) {
 if (status == google.maps.GeocoderStatus.OK) {
 var lat = results[0].geometry.location.lat();
 var lang = results[0].geometry.location.lng();
 setMarker(lat, lang);
 }
 else if (status == google.maps.GeocoderStatus.OVER_QUERY_LIMIT) {
 setTimeout(function () {
 //Recursively calling spotPosition method for lost addresses
 spotPosition(address);
 }, 1000);
 }
 });
 }

Answer 9

Here is my solution:

dependencies: Gmaps.js, jQuery

var Maps = function($) {
 var lost_addresses = [],
 geocode_count = 0;
 var addMarker = function() { console.log('Marker Added!') };
 return {
 getGecodeFor: function(addresses) {
 var latlng;
 lost_addresses = [];
 for(i=0;i<addresses.length;i++) {
 GMaps.geocode({
 address: addresses[i],
 callback: function(response, status) {
 if(status == google.maps.GeocoderStatus.OK) {
 addMarker();
 } else if(status == google.maps.GeocoderStatus.OVER_QUERY_LIMIT) {
 lost_addresses.push(addresses[i]);
 }
 geocode_count++;
 // notify listeners when the geocode is done
 if(geocode_count == addresses.length) {
 $.event.trigger({ type: 'done:geocoder' });
 }
 }
 });
 }
 },
 processLostAddresses: function() {
 if(lost_addresses.length > 0) {
 this.getGeocodeFor(lost_addresses);
 }
 }
 };
}(jQuery);
Maps.getGeocodeFor(address);
// listen to done:geocode event and process the lost addresses after 1.5s
$(document).on('done:geocode', function() {
 setTimeout(function() {
 Maps.processLostAddresses();
 }, 1500);
});

Answer 10

You can't avoid google map geocode limit : you must obtain and store LatLng informations.

If you have several city and you don't want to manually obtain LatLng, you can use a timeout to progressly call geocoder.geocode and save information inside a javascript object and then use console.log(JSON.stringify(object, null, 4)); to obtain in the console a pretty printed object, so, for example

 var locationsPosition = {
 "Johannesburg - ZA": {
 "lat": -26.2041028,
 "lng": 28.047305100000017
 },
 "Chennai - IN": {
 "lat": 13.0826802,
 "lng": 80.27071840000008
 }
 }

so now you can add marker using locationsPosition["Johannesburg - ZA"].lat and .lng

Answer 11

It sounds like you need to do one of two things:

1. change your process to cache results if you don't get that many that are unique per day; or
2. upgrade to a service that allows you to process more per day (paid) Google licensing for higher limits is quite expensive.

If you want the same level of accuracy and pay much less or even free, there are other solutions out there, including ours.

Of course, simply switching to another API isn't the ideal solution. I would instead suggest going with option 1 and optimize your service to cache. Unless you are in need of more than 2500 UNIQUE location requests daily.

Disclaimer: I work for Geocode.Farm