Python Error in NDB after python 3 upgrade

Question

I am working on some code that was recently upgrade from python 2 to python 3 that is part of google app engine.

Can I ask for some assistance tracing this? I do not know if this is because something was missing during the conversion or due to the fact sync is a seperate microservice from app which appears to have no issue using the datastore

sync.yaml:
 service: sync
 instance_class: F2
 automatic_scaling:
 max_instances: 1
 runtime: python312
 app_engine_apis: true
 entrypoint: gunicorn -b :$PORT sync:app 
 #inbound_services:
 #- warmup
 #libraries:
 #- name: jinja2
 # version: latest
 #- name: ssl
 # version: latest
 # taskqueue and cron tasks can access admin urls
 handlers:
 - url: /.*
 script: sync.app
 secure: always
 redirect_http_response_code: 301
 env_variables:
 MEMCACHE_USE_CROSS_COMPATIBLE_PROTOCOL: "True"
 NDB_USE_CROSS_COMPATIBLE_PICKLE_PROTOCOL: "True"
 DEFERRED_USE_CROSS_COMPATIBLE_PICKLE_PROTOCOL: "True"
 CURRENT_VERSION_TIMESTAMP: "1677721600"
sync.py:
collection_dbs, collection_cursor = model.Collection.get_dbs(
 order='name'
) 
collection.py:
from google.cloud import ndb
from google.appengine.api import memcache
from NdbSearchableBase import SearchableModel
from api import fields
import model
import util
import datetime
class Collection(SearchableModel, model.Base):
 slug = ndb.StringProperty(required=True, verbose_name=u'Collection URL')
 name = ndb.StringProperty(required=True, verbose_name=u'Collection Name')
 short_desc = ndb.TextProperty(verbose_name=u'Short Description')`
base.py
@classmethod
def get_dbs(cls, query=None, ancestor=None, order=None, limit=None, cursor=None, **kwargs):
 args = parser.parse({
 'cursor': wf.Str(missing=None),
 'limit': wf.Int(missing=None, validate=validate.Range(min=-1)),
 'order': wf.Str(missing=None),
 })
 return util.get_dbs(
 query or cls.query(ancestor=ancestor),
 limit=limit or args['limit'],
 cursor=cursor or args['cursor'],
 order=order or args['order'],
 **kwargs
 )
util.py
def get_dbs(
 query, order=None, limit=None, cursor=None, prev_cursor=False,
 keys_only=None, **filters
):
 model_class = ndb.Model._kind_map[query.kind]
 query_prev = query
 if order:
 for o in order.split(','):
 if o.startswith('-'):
 query = query.order(-model_class._properties[o[1:]])
 if prev_cursor:
 query_prev = query_prev.order(model_class._properties[o[1:]])
 else:
 query = query.order(model_class._properties[o])
 if prev_cursor:
 query_prev = query_prev.order(-model_class._properties[o])
 for prop, value in filters.items():
 if value is None:
 continue
 for val in value if isinstance(value, list) else [value]:
 query = query.filter(model_class._properties[prop] == val)
 if prev_cursor:
 query_prev = query_prev.filter(model_class._properties[prop] == val)
 limit = limit or config.DEFAULT_DB_LIMIT
 if limit == -1:
 return list(query.fetch(keys_only=keys_only)), {'next': None, 'prev': None}
 cursor = Cursor.from_websafe_string(cursor) if cursor else None
 model_dbs, next_cursor, more = query.fetch_page(
 limit, start_cursor=cursor, keys_only=keys_only,
 )
 next_cursor = next_cursor.to_websafe_string() if more else None
 if not prev_cursor:
 return list(model_dbs), {'next': next_cursor, 'prev': None}
 model_dbs_prev, prev_cursor, prev_more = query_prev.fetch_page(
 limit, start_cursor=cursor.reversed() if cursor else None, keys_only=True
 )
 prev_cursor = prev_cursor.reversed().to_websafe_string() \
 if prev_cursor and cursor else None
 return list(model_dbs), {'next': next_cursor, 'prev': prev_cursor}

2025年05月09日 16:39:19 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/tasklets.py", line 319, in _advance_tasklet 2025年05月09日 16:39:19 sync[20250425t195045] yielded = self.generator.throw(type(error), error, traceback) 2025年05月09日 16:39:19 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:19 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/_datastore_query.py", line 116, in fetch 2025年05月09日 16:39:19 sync[20250425t195045] while (yield results.has_next_async()): 2025年05月09日 16:39:19 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:19 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/tasklets.py", line 319, in _advance_tasklet 2025年05月09日 16:39:19 sync[20250425t195045] yielded = self.generator.throw(type(error), error, traceback) 2025年05月09日 16:39:19 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:19 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/_datastore_query.py", line 343, in has_next_async 2025年05月09日 16:39:19 sync[20250425t195045] yield self._next_batch() # First time 2025年05月09日 16:39:19 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:19 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/tasklets.py", line 319, in _advance_tasklet 2025年05月09日 16:39:19 sync[20250425t195045] yielded = self.generator.throw(type(error), error, traceback) 2025年05月09日 16:39:19 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:19 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/_datastore_query.py", line 373, in _next_batch 2025年05月09日 16:39:19 sync[20250425t195045] response = yield _datastore_run_query(query) 2025年05月09日 16:39:19 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:19 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/tasklets.py", line 319, in _advance_tasklet 2025年05月09日 16:39:19 sync[20250425t195045] yielded = self.generator.throw(type(error), error, traceback) 2025年05月09日 16:39:19 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:19 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/_datastore_query.py", line 1030, in _datastore_run_query 2025年05月09日 16:39:19 sync[20250425t195045] response = yield _datastore_api.make_call( 2025年05月09日 16:39:19 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:19 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/tasklets.py", line 323, in _advance_tasklet 2025年05月09日 16:39:19 sync[20250425t195045] yielded = self.generator.send(send_value) 2025年05月09日 16:39:19 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:19 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/_retry.py", line 112, in retry_wrapper 2025年05月09日 16:39:19 sync[20250425t195045] raise core_exceptions.RetryError( 2025年05月09日 16:39:20 sync[20250425t195045] google.api_core.exceptions.RetryError: Maximum number of 3 retries exceeded while calling <function make_call..rpc_call at 0x3ebfd2fc2700>, last exception: 503 Getting metadata from plugin failed with error: '_AppEnginePoolManager' object has no attribute 'connection_from_host' 2025年05月09日 16:39:22 sync[20250425t195045] [2025年05月09日 16:39:22,717] ERROR in app: Exception on /keepalive/ [GET] 2025年05月09日 16:39:22 sync[20250425t195045] Traceback (most recent call last): File "/layers/google.python.pip/pip/lib/python3.12/site-packages/flask/app.py", line 2529, in wsgi_app response = self.full_dispatch_request() 2025年05月09日 16:39:22 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:22 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/flask/app.py", line 1825, in full_dispatch_request 2025年05月09日 16:39:22 sync[20250425t195045] rv = self.handle_user_exception(e) 2025年05月09日 16:39:22 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:22 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/flask/app.py", line 1823, in full_dispatch_request 2025年05月09日 16:39:22 sync[20250425t195045] rv = self.dispatch_request() 2025年05月09日 16:39:22 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:22 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/flask/app.py", line 1799, in dispatch_request 2025年05月09日 16:39:22 sync[20250425t195045] return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) 2025年05月09日 16:39:22 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:22 sync[20250425t195045] File "/workspace/sync.py", line 224, in keepalive 2025年05月09日 16:39:22 sync[20250425t195045] collection_dbs, collection_cursor = model.Collection.get_dbs( 2025年05月09日 16:39:22 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:22 sync[20250425t195045] File "/workspace/model/base.py", line 33, in get_dbs 2025年05月09日 16:39:22 sync[20250425t195045] return util.get_dbs( 2025年05月09日 16:39:22 sync[20250425t195045] ^^^^^^^^^^^^^ 2025年05月09日 16:39:22 sync[20250425t195045] File "/workspace/util.py", line 196, in get_dbs 2025年05月09日 16:39:22 sync[20250425t195045] model_dbs, next_cursor, more = query.fetch_page( 2025年05月09日 16:39:22 sync[20250425t195045] ^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:22 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/query.py", line 1201, in wrapper 2025年05月09日 16:39:22 sync[20250425t195045] return wrapped(self, *dummy_args, _options=query_options) 2025年05月09日 16:39:22 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:22 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/utils.py", line 118, in wrapper 2025年05月09日 16:39:22 sync[20250425t195045] return wrapped(*args, **new_kwargs) 2025年05月09日 16:39:22 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:22 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/utils.py", line 150, in positional_wrapper 2025年05月09日 16:39:22 sync[20250425t195045] return wrapped(*args, **kwds) 2025年05月09日 16:39:22 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:22 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/query.py", line 2267, in fetch_page 2025年05月09日 16:39:22 sync[20250425t195045] return self.fetch_page_async(None, _options=kwargs["_options"]).result() 2025年05月09日 16:39:22 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:22 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/tasklets.py", line 210, in result 2025年05月09日 16:39:22 sync[20250425t195045] self.check_success() 2025年05月09日 16:39:22 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/tasklets.py", line 157, in check_success 2025年05月09日 16:39:22 sync[20250425t195045] raise self._exception 2025年05月09日 16:39:22 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/tasklets.py", line 319, in _advance_tasklet 2025年05月09日 16:39:22 sync[20250425t195045] yielded = self.generator.throw(type(error), error, traceback) 2025年05月09日 16:39:22 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:22 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/query.py", line 2310, in fetch_page_async 2025年05月09日 16:39:22 sync[20250425t195045] while (yield iterator.has_next_async()): 2025年05月09日 16:39:22 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:22 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/tasklets.py", line 319, in _advance_tasklet 2025年05月09日 16:39:22 sync[20250425t195045] yielded = self.generator.throw(type(error), error, traceback) 2025年05月09日 16:39:22 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:22 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/_datastore_query.py", line 343, in has_next_async 2025年05月09日 16:39:22 sync[20250425t195045] yield self._next_batch() # First time 2025年05月09日 16:39:22 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:22 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/tasklets.py", line 319, in _advance_tasklet 2025年05月09日 16:39:22 sync[20250425t195045] yielded = self.generator.throw(type(error), error, traceback) 2025年05月09日 16:39:22 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:22 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/_datastore_query.py", line 373, in _next_batch 2025年05月09日 16:39:22 sync[20250425t195045] response = yield _datastore_run_query(query) 2025年05月09日 16:39:22 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:22 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/tasklets.py", line 319, in _advance_tasklet 2025年05月09日 16:39:22 sync[20250425t195045] yielded = self.generator.throw(type(error), error, traceback) 2025年05月09日 16:39:22 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:22 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/_datastore_query.py", line 1030, in _datastore_run_query 2025年05月09日 16:39:22 sync[20250425t195045] response = yield _datastore_api.make_call( 2025年05月09日 16:39:22 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:22 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/tasklets.py", line 323, in _advance_tasklet 2025年05月09日 16:39:22 sync[20250425t195045] yielded = self.generator.send(send_value) 2025年05月09日 16:39:22 sync[20250425t195045] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025年05月09日 16:39:22 sync[20250425t195045] File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/ndb/_retry.py", line 112, in retry_wrapper 2025年05月09日 16:39:22 sync[20250425t195045] raise core_exceptions.RetryError( 2025年05月09日 16:39:23 sync[20250425t195045] google.api_core.exceptions.RetryError: Maximum number of 3 retries exceeded while calling <function make_call..rpc_call at 0x3ebff8fff1a0>, last exception: 503 Getting metadata from plugin failed with error: '_AppEnginePoolManager' object has no attribute 'connection_from_host'

• NoCommandLine: What exactly are you trying to do (what is your code supposed to be doing)? Where in your code are you having errors (try to narrow it down maybe by adding print statements so you know where your code stops running)
• ffejrekaburb: Thank you for the reply. I don't understand the lower layers of ndb. The error bubbles up through this call. These call support enumeration of two google data store entities. cursor = Cursor.from_websafe_string(cursor) if cursor else None line: 196 model_dbs, next_cursor, more = query.fetch_page( limit, start_cursor=cursor, keys_only=keys_only, )
• NoCommandLine: Are you saying - you want to run a paginated query (using cursors) and that's where you're getting an error?
• ffejrekaburb: Yes, that is my understanding of the execution path. I don't understand the exception that is being produced. This was working with python 2. Is there documentation available to help me understand?
• NoCommandLine: See my response (it has sample code) to one of your other questions

Answer 1

In your sync.py try wrapping all ndb operations inside ndb.Client().context() like:

from google.cloud import ndb
client = ndb.Client()
with client.context():
 collection_dbs, collection_cursor = model.Collection.get_dbs( order='name' )

You can refer to this documentation on Python 3 version of ndb client library.

Also, make sure that your service account has the proper permissions to query Datastore.

• yannco: Did this work? Any suggestions or confirmation to my solution would be appreciated.
• ffejrekaburb: Thanks, I'll try tommorrow. This format helps me digest information. Rereading I can see some of my assertions were misguided. I am suspecting this line below doesn't work in python3. I'm hoping there is a dropin replacement. I do not think it was being reached from the other microservice cursor = Cursor.from_websafe_string(cursor) if cursor else None model_dbs, next_cursor, more = query.fetch_page( limit, start_cursor=cursor, keys_only=keys_only, )
• ffejrekaburb: the thread already has context the simple call to get_dbs yields "last exception: 503 Getting metadata from plugin failed with error: '_AppEnginePoolManager' object has no attribute 'connection_from_host'"
• ffejrekaburb: "context activation" per googleapis.dev/python/python-ndb/latest