openstack/nova - nova - OpenDev: Free Software Needs Free Tools

Due to a previous commit, a client of make_subprocess is making a call
with an env parameter. As make_subprocess does not have such a
parameter, image upload fails. This patch adds the env keyword argument
to make_subprocess, and use that in the Popen call.
Fixes bug 1216267
Change-Id: I8098b484067b54c15fe95f2bf84c5890db7ebcb3

This commit adds the instance uuid to the rsync debug output in
the debug logs. The issue we were seeing is that when multiple
migrations are ongoing at the same time, it's not easy to
track the rsync progress of each migration.
Fixes: bug #1215629
Change-Id: Iec662bdc0aed1c6f54ecaa062335dc5504ac030f

Currently only used by XenAPI.
Sets GZIP environment variable before spawning tar -cz to force the
compression level. Defaults to none, which means the GZIP environment
variable is not set and the default (usually -6) is used.
Implements: blueprint image-compression-mode
Change-Id: I02e136b7215a4f2cefc259f40bde9c1b205737ed

This patch adds support for a new kind of Glance image, a specially crafted ISO
which supports iPXE booting, giving customers a means to roll their own image.
Two virt-layer modifications were needed. The first was adding configurations
for the iPXE ISO feature (network to use, boot menu, mkisofs_cmd). The second
was the ability to inject networking info into the ISO after it was
downloaded.
To use this feature, operators should enable the `ipxe_boot` image-property.
DocImpact
Implements blueprint xenapi-ipxe-iso-boot-support
Change-Id: I33acf9dfdff0a5ed9797723a142bc451348e8549

Send identity headers while uploading/downloading images from glance. Related
to:
1) Add identity headers while calling glanceclient from nova
Ife02059abbdce7920f1f408b71e9745d777fa770
2) Pass identity headers received in glanceclient to nova
Ifbef582aa4e64a2e7a46db43a9cc6cf8c3531dbd
3) Pass identity headers received by glance api to glance registry
Ie5f07ed6dfeaa8428de4f79c4d40d182328e6ab4
NOTE: since this is a change to a Dom0 plugin API, it requires the lock-step
upgrade of both the compute-manager and the dom0 plugin.
DocImpact
Change-Id: I6d5e3448d2c0acc392d18e0b88cec25cf313da5b

Recently a change was made to how/where the '_' method is imported. This
seems to have broken most xenserver plugins. These plugins shouldn't be
doing * imports anyway so I've gone in and cleaned up all places where
"from pluginlib_nova import *" was used and replaced with "import
pluginlib_nova".
Fixes bug 1207107
Change-Id: I3b2a299cf60543e0f766becfeb4740af791b3576

There are a few upload errors that glance plugin should
not retry upload on, as it will fail again. This patch changes
upload to not retry on the following conditions in addition
to the 401 error that it already handles:
- CONFLICT
- REQUEST_ENTITY_TOO_LARGE
- PRECONDITION_FAILED
- FORBIDDEN
Fixes bug 1203726
Change-Id: I538a4f9b133169b2e398e7dfcb3af58fef204f03

Previous _ was monkey patched into builtins whenever
certain modules were imported. This removes that and
simply imports it when it is needed.
Change-Id: I0af2c6d8a230e94440d655d13cab9107ac20d13c

It is possible for a token to expire during a very long image upload.
If this happens, there is little point in retrying the upload, as it
is most likely to fail again for the same reason.
Fixes bug 1199454
Change-Id: Id699135a8ec4c603671b16f0127d0b2cd2a55855

The current BitTorrent code was passing integer arguments to `make_subprocess`
which would in turn pass those to `popen`. This would cause a failure because
the semantics of the underlying `execve` function is to only accept a list of
strings.
The solution is to cast the arguments to a string before passing them into
`make_subrpocess`.
Fixes bug 1201594
Change-Id: I8372795ca032420641c449cb07bb47f9252ff4ef

Current the xenapi glance plugin does not close
httplib connections when exceptions are raised.
Note this code is run under python 2.4 so with
is not available.
Fixes bug 1197846
Change-Id: I5cfb067320e7506109c301532a95fe83aa4fd0f9

* Rewrite xenhost._run_command and xenstore._run_command to use
 utils.make_subprocess and utils.finish_process
 * Change exception raised by utils.finish_process to retain information
 needed by calls in xenhost and xenstore
Change-Id: Idcdb50bededf0acde92f1774d6752043ba8f97ce
Fixes: bug #1074087 

Change xenapi bittorrent plugin to not assume that torrent file
is under CONF.xenapi_torrent_base_url + image_id + ".torrent".
This allows for sources images from arbitrary torrent URLs.
Change-Id: Ica9462a51c6299b5affc861781f3054d18866458

Catch invalid-cross device linking exceptions raise from os.rename
and provide a more intelligible error message.
Change-Id: I22b6a145141431b6d4e4cb0176881f0cf0dd609f

If an entry is already gone from xenstore, any other xenstore-rm will
raise.
When we make calls to nova-agent, we wait for the response in Xenstore.
After a timeout period, we attempt to remove it. If the DomID has
changed (due to a Windows VM rebooting at the right time via sysprep,
etc)... or if the agent picked up the command and didn't respond in a
timely fashion, the removal attempt will fail. It would be normal for
the entry to be gone in this case.
This fixes the plugin to ignore when the entry is already
deleted.
Fixes bug 1193720
Change-Id: I8585a9f147cad3c5abe531620872b475eb4128c9

If an administrator has enabled the logging of guest
consoles on XCP/XenServer, this enables nova to return
the last MB of those logs to the user.
The management of the logs on the server is a little
tricky, and will be sorted in a later patch.
Change was based on this previous idea:
https://review.openstack.org/#/c/17959/
DocImpact
Part of blueprint xenapi-server-log
Change-Id: I23c83bcf8c648cc2714a0c78951acc29a16d5c31

* Handles the output of vhd-util check a bit more intelligently.
 * Changes it to use LOG like in other modules
 * Adds a necessary import (errno)
Change-Id: I75d6b9e081159bc29732a721deb7a557e5f2ee06

Fix for bug 1160323.
DocImpact
Depends on a version of XAPI supporting relax-xsm-sr-check.
Currently no release versions support this, so the change at
https://github.com/xen-org/xen-api/pull/1116 would need to be applied
to the source to enable this. XenServer 6.2 is expected to
support this mode, and possibly some future hotfixes for XenServer 6.1.
It also depends on a configuration change which must be documented and added to devstack:
* Set "relax-xsm-sr-check = true" in /etc/xapi.conf
This commit makes the following changes:
* Attach the SR on the destination host prior to migrate
* Returns the SR-ref from the pre_migrate call (API change)
* Populates the XS VDI map with the sr-ref on the destination host
* Removes the connection to the SR from the source host post-migrate
* Added plugin to determine if iSCSI block migration is enabled
Associated tempest test at https://review.openstack.org/#/c/26478/
Change-Id: I917d9cf094190d636f4b9e14f6c8e728ff85af0e
Fixes: bug 1160323

Fixes bug 1157695
This fix removes the parted dependency from the rpm specification file,
and also gets rid of the assumption, that the build-rpm.sh is executed
at a specific directory.
Change-Id: I1dcf9096a783a88bc09af0f4f4d5541a7e505748

With this change administrators will be able to examine hypervisor
logs for real-time progress of rsync operations (resizes, migrates).
Change-Id: I463b574be1021e141fe2c8380b9c07f6cd860c78

In 1154253 we:
 Drop gzip flag from tar command for OVF archives
While support more OVF file formats is a worthy goal the approach
in 1154253 is going to cause serious issues for existing
Nova XenServer users who are already zipping their OVS images.
I would also point out that our utils.py xenserver plugins still
zips tarballs we create (see create_tarball) so the new code
causes internal inconsistencies as well.
This should fix SmokeStack.
Fixes LP Bug #1155783.
Change-Id: Iff6db314c0dec6df2e48d98db2a547f6624b9d9a

Fixes bug 1154253
Change-Id: I3880d9b187dc25d2ff44fe502eec4d1e0bce4218

Add a retry for the xapi glance plugin to handle transient issues
when uploading the vhd.
An example issue we're seeing is a connection timeout:
['XENAPI_PLUGIN_FAILURE', 'upload_vhd', 'error',
		"(110, 'Connection timed out')"]
To work around transient issues such as a connection timeout, we
should retry based on glance_num_retries before outright failing.
Change-Id: Ice6fdd3dd39ef40e5997d69209aaafa66bff5d6e
Fixes: bug #1134493 

Update all references of "LLC" to "Foundation".
Change-Id: I009e86784ef4dcf38882d64b0eff484576e04efe

Tox and run_tests.sh were running PEP8 checks against different
file-sets. This patch refactors the logic to determine which files to
run PEP8 checks on into `tools/run_pep8.sh` where it can be called by
both tox and `run_tests.sh`.
Additional fixes:
Some of our Python XenAPI Dom0 plugins don't end in *.py but should
still be checked by PEP8. This patches fixes the hacking.py violations
in the files and adds them back to the srcfiles list.
Merged tools/unused_imports.sh into tools/run_pep8.sh
Change-Id: Id5edd1acb644ab938beffc3473494a179d9d8cda

Flagged with: https://github.com/lyda/misspell-check
Run with: git ls-files | misspellings -f -
Fixes bug: 1100083
Change-Id: Icf1f844fea8ad0a1101d1dc64b9a126608e9536e

fix the N402 errors that have slipped in in the last 48 hrs since
starting this patch series.
fix an N401 error that our scanner current can't find because it
only looks for doc strings on classes and defs.
this is the xeno's paradox of patch series, but we're getting close.
Change-Id: I4a763bb4c812335d853eae05c72464f18ab93297

When the bandwidth polling task tries to update bw usage for an
instance that does not have an entry for the last two periods already
in the bw_usage_cache table, it will throw an exception in the
polling task. This was just a misses 'if' statement.
Also:
xenserver: fix swapped bw data in xs plugin.
This fixes bug 1075255
Change-Id: I44bb143039fcdfc8dacb13b67ae8f79dc5f38777

This patch adds a new dom0 plugin which supports downloading images via
BitTorrent. Torrent metadata files are assumed to be served from a
webserver which is specified by the `torrent_base_url` config.
Under the hood, the dom0 plugins calls out to rasterbar's libtorrent via
Python bindings in order perform the initial download as well as the
seeding thereafter.
Implements BP xenserver-bittorrent-images
Change-Id: I824720a6e3a37317080a22cd7405d2a88172c3ef

RPM builds will fail until this line is added to include
the new plugin.
Fixes bug 1059772
Change-Id: I06d78cf199d4cb580003e4c220af3efc8dbe437c

This patch removes our legacy handling of swap in the image. Now that
we're generating swap on-the-fly, this stop-gap solution can go away.
Change-Id: Ied3198f77af8dabb6cfbf2ab9cfb3a4eb18e32ea

Windows can take longer than the default 30 seconds for resetnetwork
requests. Double the timeout for the command to 60 seconds, but add
a flag so it can be changed without code changes in the future.
At the same time, add a flag for all other agent requests too.
Change-Id: Iba91c37fd5596ea0dd63c20f74925972df1ca715

This changes the method used to poll xenserver for bandwidth data.
The reccomended way of collecting such data from xenserver (namely the
RRD files provided by the hosts) do not seem to be reliable, they
will sometimes be correct, often will be signifigantly under (> 10%),
and occasionally will show artifacts, such as phantom 4gb bandwidth
'spikes'.
This patch changes that to use the much simpler method of simply polling the
byte counters on the VIF network devices on the host. (We have old non-nova
code that does that on xenserver, and that method is known to work).
This should also make it much easier for other hypervisors other than
xenserver to implement bandwidth polling, as polling the counters is a rather
more universal method.
Fixes bug 1055737
Change-Id: I6a280d8bbfcc74914f888b11bc09349a270a5f58

Fixes bug 1055431.
As the scheduler wants to boot a vm_mode=xen type of image, the host's
"supported_instances" capability is used for finding a good candidate.
In the Xapi case, this field was not populated.
This fix modifies the xapi xenhost plugin, so the Xen host capabilities
are returned back to the compute node, as "host_capabilities".
On the compute side, the mentioned information is used, to extract the
"supported_instances" information.
Change-Id: I2da11ab81f74b5b52e2c30832a694470978e21b0

The dom0 plugin code had been using `pickle` for serializing input and
`json` for serializing output which was needlessly inconsistent. This
patch makes the code use `pickle`--chosen for its better handling of
`datetime` objects--for both sending and receiving data.
This patch also refactors the code so that neither the caller nor the
callee need to explicitly worry about serialization: the caller just
passes in args and kwargs, and the callee's function signature just
accepts the args and kwargs as usual.
Bonus: Removes unecessary imports
Change-Id: I3abb42eeebd8d37d67e6c26fa7bcae66d876b3ee

This introduces a new glance_api_insecure setting that can be used to
not verify the certificate of the glance server against the certificate
authorities.
Fix bug 1042081.
Change-Id: I0a9f081425854e9c01e00dfd641e42276c878c67

Along with capitalization fixes to comments in code
Change-Id: I72ddc582001f80d954ca5a121903c689f40d08d1

Communicating with the agent requires polling for a response. The
operation uses xenstore, which is lightweight, yet the interval
in between polls was 3 seconds. This would cause longer than necessary
sleeps when an instance was booting making the overall boot slower.
Change-Id: I560c05887128f1a0e29228e859cca25ded4eceec

Unlike every other agent command, the resetnetwork command would not
wait for a response. All failures were silently ignored. Change this
to at least log a message if an error occurs.
Change-Id: I40e323607b2ce50869f3bf11e4582ff83cbed1c0