openstack/nova - nova - OpenDev: Free Software Needs Free Tools

Import latest cfg from oslo-incubator with these changes:
 Add deprecated --logdir common opt
 Add deprecated --logfile common opt.
 Allow nova and others to override some logging defaults
 Fixing the trim for ListOp when reading from config file
 Fix set_default() with boolean CLI options
 Improve cfg's argparse sub-parsers support
 Hide the GroupAttr conf and group attributes
 Fix regression with cfg CLI arguments
 Fix broken --help with CommonConfigOpts
 Fix ListOpt to trim whitespace
 updating sphinx documentation
 Don't reference argparse._StoreAction
 Fix minor coding style issue
 Remove ConfigCliParser class
 Add support for positional arguments
 Use stock argparse behaviour for optional args
 Use stock argparse --usage behaviour
 Use stock argparse --version behaviour
 Remove add_option() method
 Completely remove cfg's disable_interspersed_args()
 argparse support for cfg
The main cfg API change is that CONF() no longer returns the un-parsed
CLI arguments. To handle these args, you need to use the support for
positional arguments or sub-parsers.
Switching nova-manage to use sub-parser based CLI arguments means the
following changes in behaviour:
 - no more lazy matching of commands - e.g. 'nova-manage proj q' will
 no longer work. If we find out about common abbreviations used in
 peoples' scripts, we can easily add those.
 - the help output displayed if you run nova-manage without any args
 (or just a category) has changed
 - 'nova-manage version list' is no longer equivalent to
 'nova-manage version'
Change-Id: I19ef3a1c00e97af64d199e27cb1cdc5c63b46a82

Provide i18n to some exceptions that were not
provided before.
Change-Id: Ia320ba5f12e82686c8ee7973a77f152eb4b02bd4
Signed-off-by: Chuck Short <chuck.short@canonical.com>

Moving the test-case setup and teardown code into managed fixtures
so that we can better interact with them when we start running things
in parallel.
Part of blueprint grizzly-testtools
Change-Id: I406be0a88b14c0dba2d22b4957e26a53442bbae3

Change-Id: I9c7dfa7664badcafb800c02ee608699e56b344ab
Fixes: bug #1084261 

Prevent nova-manage from stack tracing when the wrong number of
arguments are passed in.
Fixes bug 1083315
Change-Id: I7b9a237a82b4255d88c01c6a56ad0ccd4a6ee984

Adds support for a configurable set of trusted directories to search
executables in (exec_dirs), which defaults to system PATH. If your
filter specifies an exec_path that doesn't start with '/', then it
will be searched in exec_dirs. Avoids having to write multiple
filters to care for distro differences. Fixes bug 1079723.
Also returns a specific error rather than try to run absent executables.
Change-Id: Idab03bb0be6832a75ffeed4e78d25d0543f5caf9

Updates the rxtx_factor validations in the instance_types modules so
that rxtx_factor is validated as a float. Given that rxtx_factor is stored
as a float in the database this makes sense... and also adheres
to some of the extension documentation as well (although some of
the extension docs were incorrect as well).
Previously rxtx_factor was being cast into an int which caused it to
be stored and displayed incorrectly via the API.
This patchset adds a test which fails with the old code.
Additionally some of the API docs are corrected so that rxtx_factor is
listed as a float in all examples.
Fixes LP Bug #1081287.
Change-Id: Iae98522a1f2ed63cbd2497b0b0af5ac2d9bb935c

nova-dhcpbridge's FLAGFILE environment variable is one of the few
remaining references we have to "flags".
Switching to use a new environment variable is easy, but we need
to take care that when you upgrade the code the compute and network
services might not be restarted immediately.
If this happens, the services may launch a new dnsmasq (so rootwrap
needs to continue to support FLAGFILE) or an existing dnsmasq might run
the lease script (so the script needs to support FLAGFILE).
Change-Id: Ieb55daa2507abb8327c7f5a2ef2f682740f31a5d

We always launch dnsmasq with FLAGFILE set, so something is very broken
if FLAGFILE isn't set when nova-dhcpbridge is launched.
Also, the fallback to CONF.dhcpbridge_flagfile is a little strange - at
this point, we haven't parsed any config files so the only possible
value of this config option is its default value.
There's no real need to gracefully handle this condition either - a
KeyError is a straightforward enough error.
Change-Id: I734d7b739f17c8a9ab48d8ba13baacff3f00e1c4

The only reason for importing nova.config now is where one of the
options defined in that file is needed. Rather than importing
nova.config using an import statement, use CONF.import_opt() so
that it is clear which option we actually require.
In future, we will move many options out of nova.config so many
of these import_opt() calls will either go away or cause a module
other than nova.config to be imported.
Change-Id: I0646efddecdf2530903afd50c1f4364cb1d5dce1

Modules import nova.config for two reasons right now - firstly, to
reference nova.config.CONF and, secondly, if they use one of the
options defined in nova.config.
Often modules import nova.openstack.common.cfg and nova.config
which is a bit pointless since they could just use cfg.CONF if
they just want to nova.config in order to reference CONF.
Let's just use cfg.CONF everywhere and we can explicitly state
where we actually require options defined in nova.config.
Change-Id: Ie4184a74e3e78c99658becb18dce1c2087e450bb

This adds a new service called "nova-conductor" which will provide
services for nova-compute, such as completing database updates and
handling long-running tasks.
Right now, this just adds a fraction of the database-related work
so that we can proceed with incremental changes to nova-compute.
For compatibility/flexibility, both LocalAPI and (RPC)API classes
are provided. Users concerned with performance or compatibility
who wish to maintain compute-node-local database accesses can
set the [conductor] use_local=True, which will mimic the legacy
behavior.
DocImpact
Change-Id: Ib81d879f40b7daceeffad5af0631c7db68168661

nova.flags is no longer needed except to load nova config options shared
across multiple daemons.
This removes nova.flags from the bin programs and makes sure that
nova.flags is imported in nova.config for now.
Change-Id: If066ac0070387bee4b41e6a78ad972f7a0955c75

Fix a couple hacking violations in Nova's bin scripts.
Change-Id: I5cab72a5a94f761f429ad45f94a95cd4bc7e30e8

* Remove volume calls from nova.db.api
* Remove volume calls from nova.db.sqlalchemy.api
* Remove tables from nova/db/sqlalchemy/models.py
* Remove Xen Storage Manager Volume Driver code
* No migration to preserve data in unused tables
* Remove quota support for volumes
DocImpact
part of bp delete-nova-volume
Change-Id: I2c82c96d67f3746e5de28f917dd8ceb0c8037e27

Fixes Bug: #1074717
During volume attach on Windows a process is spawned. The process
currently fails with the following exception, due to eventlet monkey
patching of the os module.
NotImplementedError: set_nonblocking() on a file object with no
setblocking() method (Windows pipes don't support non-blocking I/O)
The fix consists in avoiding monkey patching the os module in
nova-compute on Hyper-V
Change-Id: Ib513bc218da83b751439bad114baac24b08e3d02

Use the global CONF variable instead of FLAGS. This is purely a cleanup
since FLAGS is already just another reference to CONF.
We leave the nova.flags imports until a later cleanup commit since
removing them may cause unpredictable problems due to config options not
being registered.
Change-Id: Id0c59b2dc8002ec89ccbc5e5d7986fb68d3a693d

The cfg.ConfigOpts class has an equivalent method, so lets use that.
Change-Id: I5860230336d00d7531a0ffd255c766ff77661625

The flags module will eventually be removed and this is a first step
towards that.
Change-Id: I729b08900e53e2ae6db10633dcff3be59720fa6f

nova-manage is now getting pep8 scanned, so these fixes
are needed to make things happy.
Change-Id: I68b1d17fc06c2b2580907eb950b6087e499d65f9

This removes the majority of the nova-volume code from the codebase.
It updates relevent config options to default to cinder. It updates a
number of existing tests that were depending on code that was removed.
A few things still need to be removed:
 * volume/driver.py & volume/iscsi.py
 These files are used by the libvirt volume driver tests. These
 tests should be updated to mock the relevant calls.
 * scheduler/simple.py & scheduler/multi.py
 These files should no longer be necessary so they can be removed
 in a subsequent patch
 * exception.py cleanup
 Once the above files are removed there are a number of unused
 exceptions which can be removed
 * database calls and database tables
 The database calls have not been removed and the tables have not
 been dropped. This can be done in a separate migration
 * additional config options and nova.conf.sample
 There may be a few extra config options that can be removed and the
 conf sample can be regenerated
Implements bp delete-nova-volume
Change-Id: I0b540e54dbabd26901a7530035a38583bb521fda

nova-dhcpbridge can read in an alternate config file location but wasn't
using it to parse config options. This fixes a regression introduced in
00786bc554.
This change is motivated by the fact that nova-dhcpbridge is passed to
dnsmasq in restart_dhcp() in nova/network/linux_net.py, and there is
currently no support for overriding config options for nova-dhcpbridge
in that call. There is support for overriding the FLAGFILE env variable
in that call, but nova-dhcpbridge was ignoring that. If I start
nova-network with an alternate config this provides a way for me to pass
that config to nova-dhcpbridge as well.
Change-Id: I002ea768500555a089bd1d5b0fea2354b27fba14

Replace hardcoded topic strings like 'volume' or 'compute'
with config constants like FLAGS.volume_topic, etc. See
bug #1057831 and bug #1061628.
Change-Id: I817ecc3cbe3245b51a0c047be58d17edfec8a838

Ensure visible message in nova-volume-usage-audit go through translation.
Fixes LP: #1068946
Change-Id: Iaffe415b709dc0f515d32cad46c3fa851321fd31

utils.default_cfgfile have been removed from nova/utils, so
remove it here as well.
Fixes LP: #1065069
Change-Id: I71f6c7850123ef0c1ef4fedf608de13b31dafb32
Signed-off-by: Chuck Short <chuck.short@canonical.com>

fixes Bug #1056910
Allows the user to create a flavor without optional arguments.
The variable, ephemeral, is not currently set to a default value.
This none type value causes exceptions in subsequent code, which
occurs when specifying only mandatory parameters.
Change-Id: Ic440136b000dc296bef3bb6f8b0856ff5e7e4e79

The nova-novncproxy service was using rpc.call directly when it should
have been going through the rpcapi client proxy class. This patch fixes
that.
Fix bug 1067858.
Change-Id: I626fb01ff61ff507520d528122358ef687d297b7

This review wraps some strings which were missing out on
internationalization. It also adds a slightly more helpful error
message for quota key validation.
Change-Id: Id6f1f2578b78815b69cf43823d83ad24ca4123aa

This patch adds versioning support for the nova-network rpc API. During
Folsom development I added versioning to all rpc APIs with the exception
of network and volume. I was holding off on those to see what the fate
of that code in nova was going to be for sure. Adding it to the volume
API in nova doesn't make much sense at this point (but doing it in
Cinder does). Since nova-network is sticking around for a while, it
seems worthwhile to add a version number to the API for any future
changes while the code is still in the tree.
There are plenty of things in this rpc API that could be cleaned up or
improved, but I held off on doing any of those things. I would
recommend the same for anyone else (unless it's a bug that needs to be
fixed). I think any further improvements should be focused on Quantum
integration with Nova or Quantum itself.
Fixes bug 1060197.
Change-Id: I63ce57657388166544202af9c44c2298ae551aea

nova-manage doesn't validate the key value supplied to
update the quota, as a result unnecessary records will be
created in db and user will be under the impression that
quota value got updated.
This patch validates the input value given to the key.
fixes bug 1064359
Change-Id: I9928f30881aa2780a23005b5f69aa67a44f314c5

Also remove unused imports.
Change-Id: I6a5379ef9493fe39ba9442e809d0aeeaa97fb7b1

Removes config file converter: nova-manage config convert
Change-Id: I146bff9ac00a4885e7054f999ae52159da3f86ef

Following dprince's change to use
FLAGS, rather than cfg.CONF,
zmq_opts needs to be registered explicitly.
Change-Id: I66bc71563435088ee9224fdd603602e35f3e3bff

Python ignores SIGPIPE on startup, because it prefers to check every
write and raise an IOError exception rather than taking the signal. Most
Unix subprocesses don't expect to work this way. This patch (adapted
from Colin Watson's post at http://tinyurl.com/2a7mzh5) sets SIGPIPE
back to the default action for nova.utils.execute and nova-rootwrap
created subprocesses.
Fixes bug 1053364
Change-Id: I17e1629bb4ef4268515c6734ddb6e12746739c52

Updates quota value to -1 in db rather than None
if the user specify the quota limit as "unlimited".
Fixes bug 979087
Change-Id: I0ec412189ad9630c4a875655294c1e77886108f5

The nova-manage volume commands weren't updated when the switch
 from int volume-id's to uuid's was made. This updates the
 param2id method to call the appropriate conversion, and doesn't
 attempt to cast the value to an int.
 There is an additional issue with the nova-manage commands with the
 introduction of Cinder. The calls in nova-manage are direct to the
 nova database which when using Cinder is not valid. Add an error message
 when trying to use delete or re-attach commands as they don't work.
 This change deprecates the nova-manage delete function as it's now
 implemented as an admin extension (nova commit id: If795599d and
 cinder commit id: I29f4b892).
 The reattach command is only valid for nova-volumes, a similar extension
 will need to be added to cover cinder at which time the nova-manage volume
 reattach can be deprecated as well.
 Partial fix for bug #1051603
Change-Id: I666d4e627dee4a2025f7135560ee36c40f4bf17a

Add partially written manpages for:
* nova-all
* nova-api-ec2
* nova-api-metadata
* nova-api-os-compute
* nova-api-os-volume
* nova-api
* nova-cert
* nova-compute
* nova-console
* nova-consoleauth
* nova-dhcpbridge
* nova-network
* nova-novncproxy
* nova-objectstore
* nova-rootwrap
* nova-rpc-zmq-receiver
* nova-scheduler
* nova-volume-usage-audit
* nova-volume
* nova-xvpvncproxy
Change-Id: I3734831ce2f6b5d765e98b3f50fe8c1ad7965685

This is similar to the change to improve bulk creation speed. Before:
./bin/nova-manage floating delete --ip_range 10.250.0.0/16
337.20user 10.22system 44:07.10elapsed 13%CPU (0avgtext+0avgdata 119536maxresident)k
400inputs+0outputs (4major+11704minor)pagefaults 0swaps
After:
./bin/nova-manage floating delete --ip_range 10.250.0.0/16
6.95user 0.16system 1:04.04elapsed 11%CPU (0avgtext+0avgdata 120016maxresident)k
0inputs+0outputs (0major+11750minor)pagefaults 0swaps
Resolves bug 1023560.
Change-Id: If4f23222025cae88576f469022e0f7d907c9c4fa

When creating a new flavor, it's now possible to omit flavorid. In this
case it will be automatically generated using utils.gen_uuid().
nova-manage and OS API updated accordingly.
Fixes: bug #1043410
Change-Id: Ibf9229599dac953177fbf1ffac5242ed716142db