openstack/nova - nova - OpenDev: Free Software Needs Free Tools

Change-Id: Ibdad9607e4a046cb0c78b68ae4255679e17ff218

This reverts commit df44a7e25d.
Not only did it not account for the object versioning changes
required, or all the other string attributes we have, but the
unicode "fix" was backwards anyway (we should be decoding, not
encoding there).
Change-Id: I2369a79944fa5cb3b076e09a92f5075ec3427f46

Using the metadata saved on instance creation, we can now get all the
image related metadata we need from the instance itself.
This patch replace the logic for getting the image metadata on some
actions that shouldn't fail when the image is not accessible (create
an snapshot, resize, migrate, rescue an instance or attach an
interface).
Fixes bug 1039662
Change-Id: Id8c2f13ca96908317d271a84fd685104b64f87e9

This patch adds a couple of utility functions that enclose all the logic
for getting and parsing the image metadata stored in the instance's
system metadata.
First, this will try to fetch the metadata from the real image and will
prevent it from failing if it is not available. It will be then merged
with the image metadata stored during the instance creation.
Related to bug #1039662
Change-Id: I2130caf19858585571b1199e27f0a98ad5f08701

Fixes bug #1213080 and implements blueprint condutor-workers.
Make it easy to launch a bunch of conductor processes on a host.
Deploying multiple conductor workers per host avoids serialization on
database accesses caused by libmysqlclient.so blocking eventlet's
single thread. In an experiment on a 24-core machine, when creating 20
VMs in parallel, maximum creation time was reduced by approx. 10s when
using 20 conductor processes vis-a-vis a single conductor process.
Profiling showed that all of the savings came from faster calls into
nova.db.sqlalchemy.api.
Note that there are alternative methods for preventing the eventlet
thread from blocking during database calls. However, none of these
alternatives performed as well as multiple nova-conductor processes.
 * Instead of using the native database driver like _mysql.so, you
 can use a pure-python driver, like pymysql by setting
 sql_connection=mysql+pymysql://... in the [DEFAULT] section of
 /etc/nova/nova.conf, which eventlet will monkeypatch to avoid
 blocking. The problem with this approach is the vastly greater
 CPU demand of the pure-python driver compared to the native
 driver. Since the pure-python driver is so much more CPU
 intensive, the eventlet thread spends most of its time talking to
 the database, which effectively the problem we had before!
 * Instead of making database calls from eventlet’s thread, you can
 submit them to eventlet’s pool of worker threads and wait for the
 results. Try this by setting dbapi_use_tpool=True in the
 [DEFAULT] section of /etc/nova/nova.conf. The problem I found
 with this approach was the overhead of synchronizing with the
 worker threads. In particular, the time elapsed between the
 worker thread finishing and the waiting coroutine being resumed
 was typically several times greater than the duration of the
 database call itself.
Change-Id: I8698997d211d7617ee14a1c6113056a694d70620

Override the base class implementation of instance_exists
for efficiency. This saves a lot of calls to XenAPI.
Change-Id: I07d2f136a12c08cbbc24bf3a3cf72235aed1b232

Change-Id: Ibf423f14a5c37aa298b2115bfd4936f660c6f530

Only direct dependencies need to be listed in requirements.txt.
pyparsing was previously added to workaround installation ordering
issues, which is no longer necessary.
Change-Id: I45bdd1d0dabfcc7c80314c28072950cfc5a59615

Conversion type is missing in some places which would cause some
unexcepted error. By using 'grep -rn "%(\w\+)[^\%a-z0-9\.\-]"',
we could find all cases of '%(variable_a)' and fix them.
Change-Id: Id4b376a92ad23db2cb5380be85d4fe937f1a5cb7
Closes-Bug: #1221026 

This patch makes the necessary changes in the libvirt driver to enable
it to use some of the features of the new block device mapping format.
After this patch it will be possible to set the bus, and device_type per
block device, and libvirt driver will honor these when spawning an
instance (note that attaching a volume still does not use the new data
format).
It utilizes some of the existing code in the blockinfo module to be able
to default device names (it does so by overriding the methods introduced
in I84541f8ff6e1b5978734e5def69946d014c66fdf), and also assign default
values to fields like device_type and disk_bus if it is not provided or
if it is bogus. As this implies the driver changing the block devices in
the database, a new virtapi method block_device_mapping_update is added
to accommodate this. Some of the libvirt specific code paths in the
general defaulting function in compute utils have been removed as they
are not needed since the driver now takes care of this.
Further to that, this patch modifies some of the code paths in the
libvirt driver that use the block device info directly (and not through
the blockinfo module) to be aware of the new format. Due to very nicely
factored code - there were only a few instances of this in the driver
itself.
It also overrides the libvirt driver's need_legacy_block_device_info
method to tell the compute manager to feed it the new format when
needed.
This patch concludes the blueprint: improve-block-device-handling
Change-Id: I8efd6af6706a097fb540e040a86ccbeaf131631f

This patch removes the check in the API that prevented having block
devices without 'device_name' and makes the compute manager capable of
guessing those names.
Before creating the instance, the manager will try to guess a valid name
using the method 'default_device_names_for_instance', which should be
implemented by the drivers, or defer to the generic function from the
utils module 'default_device_names_for_instance' which uses the already
existing 'get_device_name_for_instance' function.
It also takes care of assigning the correct device name to the root
device (boot_index=0) to make it agree with the field `root_device_name'
from the instance. If none were supplied - it will try to guess it by
calling driver 'default_root_device_name' or defer to the generic
'get_device_name_for_instance'. if driver does not provide it.
What is worth noting also is that, should the drivers override the
two methods mentioned above, the code should expect to be passed the
standard block device mapping format, and not the one used in other
driver methods.
DocImpact
Part of blueprint: improve-block-device-handling
Co-authored-by: Xavier Queralt <xqueralt@redhat.com>
Change-Id: I84541f8ff6e1b5978734e5def69946d014c66fdf

Add a temporary nova.notifier.Notifier helper class which translates
oslo.messaging.Notifier compatible calls into openstack.common.notifier
compatible calls.
This allows us to port the notifier code over to the oslo.messaging API
before actually switching over oslo.messaging fully.
This patch contains no functional changes at all, except that all
notifications go through this temporary helper class.
Some notes on the new API:
 * The notifier API is changed so that what was previously global state
 is now encapsulated in a Notifier object. This object also includes
 the publisher_id and has error()/info()/etc. methods rather than
 just notify().
 * The notify_decorator() helper wasn't carried across to the new API
 because its semantics are a bit weird. Something along these lines
 could be added in future, though.
 * We use a fake Notifier implementation for tests because there's no
 API in oslo.messaging to actually get the notifications queued
 up in the fake notification driver, which is a bit dumb. However,
 this feels like the right thing to do anyway. We're not wanting
 to test oslo.messaging.Notifier itself, but rather we want to test
 how we call it.
blueprint: oslo-messaging
Change-Id: I262163c7e05e6a6fb79265e904ce761fc3ac5806