[openstack-dev] [magnetodb] Using gevent in MagnetoDB. OpenStack standards and approaches
Dmitriy Ukhlov
dukhlov at mirantis.com
Tue Mar 18 18:51:12 UTC 2014
Hello openstackers,
We are working on MagnetoDB project and trying our best to follow
OpenStack standards.
So, MagnetoDB is aimed to be high performance scalable OpenStack based
WSGI application which provide interface to high available distributed
reliable key-value storage. We investigated best practices and separated
the next points:
1.
to avoid problems with GIL our application should be executed in
single thread mode with non-blocking IO (using greenlets or another
python specific approaches to rich this)
2.
to make MagnetoDB scalable it is necessary to make MagnetoDB
stateless. It allows us run a lot of independent MagnetoDB processes
and switch all requests flow between them:
1.
at single node to load all CPU's cores
2.
at the different nodes for horizontal scalability
3.
use Cassandra as most reliable and mature distributed key-value storage
4.
use datastax python-driver as most modern cassandra python client
which supports newest CQL3 and Cassandra native binary protocol
features set
So, considering this points The next technologies was chosen:
1.
gevent as one of the fastest non-blocking single-thread WSGI server.
It is based on greenlet library and supports monkey patching of
standard threading library. It is necessary because of datastax
python driver uses threading library and it's backlog has task to
add gevent backlog. (We patched python-driver ourselves to enable
this feature as temporary solution and waiting for new python-driver
releases). It makes gevent more interesting to use than other
analogs (like eventlet for example)
2.
gunicorn as WSGI server which is able to run a few worker processes
and master process for workers managing and routing request between
them. Also it has integration with gevent and can run gevent based
workers. We also analyzed analogues, such as uWSGI. It looks like
more faster but unfortunately we didn't manage to work uWSGI in
multi process mode with MagnetoDB application.
Also I want to add that currently oslo wsgi framework is used for
organizing request routing. I know that current OpenStack trend is to
migrate WSGI services to Pecan wsgi framework. Maybe is it reasonable
for MagnetoDB too.
We would like to hear your opinions about the libraries and approaches
we have chosen and would appreciate you help and support in order to
find the best balance between performance, developer friendness and
OpenStack standards.
--
Best regards,
Dmitriy Ukhlov
Mirantis Inc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140318/e95f38e5/attachment.html>
More information about the OpenStack-dev
mailing list