openstack/nova - nova - OpenDev: Free Software Needs Free Tools

This adds a non-voting nova-multi-cell job which
is a 2-node job with 2 non-cell0 cells (cell1 and cell2).
The compute on the controller will be in cell1 and the
subnode compute will be in cell2.
Until cross-cell resize is supported, cold migration
has to be disabled since we can't cold migrate to
the same host (and there is only one compute host per
cell). As a result, resize to the same host is enabled
(it is normally disabled in the tempest multinode job).
The parent job enables live migration testing but since
we only have one host per cell and don't support live
migration across cells, the live migration tests are
also disabled.
The job is also setup to run with python3 since that
is the way of the future.
Non-essential services, like cinder-backup, are disabled
since we (1) won't need it for the tests we run and
(2) would like to save as much room on the hosts as possible
since we're running multiple cell conductors on the
controller host.
Since this is new and its already uncovered at least one
bug (bug 1826472), the job is non-voting in the check queue
for now until we're happy with its stability.
Related to blueprint cross-cell-resize
Change-Id: I8ad3f8bcbecf0f6b1d0f57e9f61ff21bd7fbe365

Thorough replacement of git.openstack.org URLs with their opendev.org
counterparts.
Change-Id: I3e0af55e0707f04428a422b973d016ad30c82a12

We're going to start ripping out cells v1. First step is removing the
tests for this soon-to-be-removed feature.
Part of blueprint remove-cells-v1
Change-Id: I2031bf2cc914d134e5ed61156d2cfc621f006e0b
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>

This doesn't appear to be used by any job, in-tree or out-of-tree.
Remove it.
Change-Id: Iee89cf53069224fa24530c084639cb7788f8d284
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>

The CachingScheduler has been deprecated since Pike [1].
It does not use the placement service and as more of nova
relies on placement for managing resource allocations,
maintaining compabitility for the CachingScheduler is
exorbitant.
The release note in this change goes into much more detail
about why the FilterScheduler + Placement should be a
sufficient replacement for the original justification
for the CachingScheduler along with details on how to migrate
from the CachingScheduler to the FilterScheduler.
Since the [scheduler]/driver configuration option does allow
loading out-of-tree drivers and the scheduler driver interface
does have the USES_ALLOCATION_CANDIDATES variable, it is
possible that there are drivers being used which are also not
using the placement service. The release note also explains this
but warns against it. However, as a result some existing
functional tests, which were using the CachingScheduler, are
updated to still test scheduling without allocations being
created in the placement service.
Over time we will likely remove the USES_ALLOCATION_CANDIDATES
variable in the scheduler driver interface along with the
compatibility code associated with it, but that is left for
a later change.
[1] Ia7ff98ff28b7265058845e46b277317a2bfc96d2
Change-Id: I1832da2190be5ef2b04953938860a56a43e8cddf

The libvirt driver doesn't support resize for lvm-backed instances
so we need to skip this test to get the nova-lvm job to pass again.
Change-Id: Id752b539babadd187b4c999039cc4ca655437d47
Closes-Bug: #1782607 

This skips a couple more tempest rebuild tests for
latent intermittent rebuild race failures
due to status sync delays with cells v1.
Change-Id: Ib2dcbba7f447f54c36877a4e7c29d1b6839a0a80
Related-Bug: #1709985 

This changes the nova-lvm job to run in the check queue on libvirt
driver changes only, and only runs the tempest compute API tests
to save time since we don't need to run things like the cinder,
glance, neutron etc API tests.
Once we're comfortable with the stability of this job we can
make it voting and gating.
This is in response to bug 1771700 which could have been prevented
if we were gating on the nova-lvm job on libvirt changes.
Change-Id: Ieaf00bcb6cb885e544d05b6f7276b6470b123258
Related-Bug: #1771700 

Another rebuild test that intermittently fails the cells v1
job because of latent races with status changes in cells v1.
Change-Id: Ic422a5d7ac795e6e6882f1f0ad82022a7bd42229
Related-Bug: #1709985 

Tempest change I48f62d36a871a95601ebf277a2f39a41a873daa8 added
a new rebuild test for microversion 2.63 so we need to exclude that
test also for cells v1 since rebuild is racy in cells v1.
Change-Id: Iff89b9714e2413716bf87db6f0d773787c06eda3
Related-Bug: #1777478 

This adds yet another rebuild test to the blacklist for the
cells v1 job. Rebuild status updates are racy in the cells v1
job because of the async status updates.
Change-Id: I42f08ee2a7282c9cad761bbe0daa111e79678791
Related-Bug: #1709985 

This is another occurrence of a rebuild test randomly timing out
waiting for status changes in the cells v1 job, so blacklist it.
Cells v1 is deprecated and should be gone soon anyway, so we don't
need to waste time hitting stuff like this.
Change-Id: Icb1d33c6e602467e21efe4838cb6edbadab14834
Related-Bug: #1709985 

Now that the nova-cells-v1 job is running with neutron
we can remove several explicit blacklist entries for things
that wouldn't work with nova-network and cells v1.
To be clear, these tests won't be added back for nova-cells-v1,
it's just that with the configuration on that job the skips
are handled via config in tempest rather than our blacklist rc
file, which is cleaner.
Depends-On: https://review.openstack.org/549785/
Change-Id: Ib61f159056d229ad190d3f3b61e6d7e3eb6d067f

This adds an experimental queue job to run non-slow tempest
API and scenario tests using the CachingScheduler as the
scheduler driver.
A blacklist is added since there are a few tests that rely on
filters which don't work well with the CachingScheduler.
The CachingScheduler is deprecated, but this is useful to have
when we're making changes to the scheduler or flows within the
code that involve the scheduler, such as the alternate hosts
work.
Change-Id: I8630ea11c3067ed934de2ef27a63432418e98c33

This is no longer used since I94e2898914accea0e973db3788370d70402e75a7.
Change-Id: I3206103a861e38c69c472d41b243f947c5910b0f

The cells API doesn't route the os-server-external-events API
and this test relies on that working, so we have to blacklist it.
Change-Id: I92e316cb9cfa5d47c415ba06edf45d7de68677f4
Closes-Bug: #1721644 

These are other rebuild tests that randomly fail the
cells v1 job waiting for state changes due to VIF
races.
Since cells v1 is deprecated, let's just skip this.
Change-Id: Ia00015a8cbb930efd274830b69f296a257578700
Related-Bug: #1709985 

This test randomly fails due to a timeout in cells v1
jobs and is a latent issue. Since cells v1 is deprecated
and we aren't fixing latent bugs, let's just skip this.
Change-Id: I386df03f406dd0f1847a0d091e070df7786f616e
Related-Bug: #1709985 

The Xen toolstack is trying to locate the kernel by running pygrub on
the host (as opposed to in guest) on the disk image, this fails for
encrypted volume.
Skip the Tempest test until this is resolved.
Change-Id: I173ac0ae8ebfb249d75a29e425e90c8928c3da40
Related-Bug: 1702897

I187e560911f5d5d482eb7959e5174068c4c9a801 enabled
ssh validation by default which uses floating IPs
by default, which isn't support with cells v1.
Since cells v1 is on life support we're going to
just disable ssh validation in this job.
Change-Id: I33f827cf76c7d99a599f9c0c509453cf2af6c2a6
Closes-Bug: #1693949 

https://xenbits.xen.org/gitweb/?p=openstack/ci-loop-config.git;a=commit;h=72110826f843cfd1b6b55eba5c4f41b6295ca892
Change-Id: I170647d41e6985ff01b1a42fbdc4b1bd8f577ef7

test_stamp_pattern was recently unskipped:
I2c13099595d8ee8099e02e3f06096078c7d27c34
It uses security groups which cells v1 does
not support, so we have to skip this test in
the cells v1 job.
Change-Id: Id7e8351026ca86694fc9814002b96260f6ba10f9
Closes-Bug: #1664607 

Similar to the RC files for excluding tests in the cells v1
and lxc jobs, this is for an LVM job that's being created.
This list was tested in this change:
Ic511246b36df776c242f0dbf86fe0f6e79ef9e0a
Related to blueprint libvirt-instance-storage
Change-Id: I90cb85a503f95741b8e4f4f2147798d9414ce4ea

Tempest has a feature toggle for rescue tests which
devstack now disables for cells v1 testing, so we
can remove the regex for the rescue tests in our
blacklist file.
Change-Id: Iab0616d1b13f6a08cbc6131d2b1a182fa720f21a
Depends-On: I234f778c9d264ee2e5fe6050ca6b33283be464f7

Currently, nova blacklists tests by name. If a name changes,
the regex will not match, and the test will be attempted.
If the test idempotent ids are used instead, the test
will always be skipped.
Change-Id: Iaf189c42c342b4c2d7c77555980ed49914210bf6
Related-bug: 1562323
Closes-bug: 1562945

Tempest change 9bee3b92f1559cb604c8bd74dcca57805a85a97a
renamed a test in our blacklist so update the filter to
handle the old and new name.
The Tempest team is hesitant to revert the change so we
should handle it ourselves and eventually move to using
test uuids for our blacklist, but there might need to
be work in devstack-gate for that first.
Change-Id: Ibab3958044c21568d7fbbe0a298bb40bbbc20df3
Closes-Bug: #1562323 

Change 022802997c fixed evacuate/rebuild
in the cells API so enable the rebuild-related tests we were skipping
before.
Related-Bug: #1445629
Related-Bug: #1445631
Related-Bug: #1552046
Change-Id: I15f867d8c6ff280bc68a6abf8b0da6e970c2bb39

This reverts commit 674531df15
The test was breaking more than cells, it was breaking all
shared-storage backend jobs, like ceph, so it was reverted
here:
bbeab27ab8f05ad1287907222fb6b02990021885
Since the test no longer exists, we can remove it from the
blacklist.
Change-Id: Iee028a56f6a27d97011f26966b3584770d5287db

This commit adds a newly added tempest test to the cells exclude list.
The test is always failing on cells and it's because it's doing
operations that don't work with cells turned on. So lets exclude it
for now. Ideally all this skip logic will be in the tempest config and
we don't have to do this anymore.
Closes-Bug: #1543253
Change-Id: Ic9db51a41f95b0d18f97745a0da7e99fdfa21e51

This patch is intended to get the LXC experimental gate job
passing consistently and at least get some reliable and
regular testing of the LXC Nova capabilities.
The skipped tests pass with 3.13.0-58-generic and current
kernel HEAD, however, they do not pass with 3.13.0-76-generic
that is used for our gate testing.
Related-Bug: #1536280
Change-Id: I422094e49c498368f6eec16207379139fdc2936a

The dependent change disable resize tests in Tempest when running cells
and using non-default flavors (which are not in the cells API database).
So we can remove the skips from the regex for the cells test runs.
Depends-On: I20f46024e45e32c60275703a193a56ae8cfe7eca
Change-Id: Ib5d36c4300a84b48faecb2f8181ac0057d4bb4b8
Related-Bug: #1513925 

The new test_schedule_to_all_nodes test fails in some epic ways in
cells. Not really worth debugging, this is a new test and had just
exposed another cells / devstack issue. Just skip the test.
Change-Id: Icc71e36f4ecb015dff9e806caacd31262f7e17f7

These scenario tests run with neutron and rely on assigning floating IPs
which cells v1 doesn't support, so skip them.
Longer term I think we'll have a feature toggle in Tempest to allow
disabling these tests via config but this is quicker and easier for now
to get a cells v1 + neutron job in the experimental queue.
Change-Id: Ieb3b117449f56810f1a41fcf43e737ae919b8945

The current regex contains a lot of slightly off regex
definitions. Most of them don't hurt, and do still match the correct
way, but they make people believe that things like the * at the end of
matches is important (which it is not).
This cleans this up, plus adds a few comments to help people going
forward.
Change-Id: Ica7fc32fd5a756214db987b95d2305bc74cfb6f1

The cells architecture of doing db replay makes this call
fundamentally racey for cells. The contract is that if you request an
action on a server and it is locked, you get a 409 at the API, and are
done. If you get a 2xx, the resource is not locked, and life is good.
Except in cells, the API db will be updated, and the API will return
all is well. Except the db replay hasn't happened for the cell, so the
additional check in the nova compute for locked blows up.
The fix probably involves the nova-compute check doing a timed retry
under cells because of replication lags.
Change-Id: Ic18407b8372c286d8e258860f3e2bf76eceef2aa
Related-Bug: #1466696 

Since there are some race conditions with the EBS image checking test,
it's better to remove the entire Tempest class tests just until we
figure out what's wrong with the boot from volume stuff
Change-Id: Ice6a718b6d7b27505696a3c55fc02d44d4b187a7
Related-Bug: #1489581 

This is the test exclusion list that will be used by the new
*tempest-dvsm-lxc job.
Co-Authored-By: Matt Riedemann <mriedem@us.ibm.com>
Change-Id: Ic7ad85958ef0ef6576a868b9c4485868523c8e52

The conductor service of compute host will try to find ssh key in child
cells when creating instance. But the ssh key was only stored at parent
api cells. Therefore the conductor service will throw ssh key not found
exception.
There are three solutions:
1. add keypair_type into the instance object, along side keypair_name, etc
2. Sync ssh keypair to every child cell database
3. consider sending a message to the parent cell to fetch the keypair
This commit prefer to use third solution. Because of the data ssh
keypair should be global. And cells v2 split key_pairs table into global
API tables.
Change-Id: I156a1c3cf3e31f34cea5e240b14a9575e9e45239
Closes-Bug: #1443816 

Ie4ffd458456d03b0b817b01bbed391f359240db2 changed some TestCases names with
the consequence that those tests were becoming not excluded and consequently
the cells job was failing.
Change-Id: I81ae2918355a02c4e20470a924538d2c7fbafcf6
Closes-Bug: #1492255 

nova-cells does not support the shelve feature.
Since the nova-cells shelve tests are now disabled using
the tempest feature config options, we can remove the
shelve tests from the blacklist.
Change-Id: Iad51bfcff49b7eabc46101bd7f956adfa5958ced
Closes-bug: 1491152
Depends-On: Ibf1f9aaa63e5f17b7d8774b511940ba8421e0887

We need to whittle down the large regex of excluded tests for the
tempest-dsvm-cells job but we also need to be able to test changes to
the regex, like removing entries that we think are now working.
This change moves the regex from project-config in the
tempest-dsvm-cells job definition to nova so that nova can control and
test the changes on it's own, since project-config is not currently able
to self-test job changes.
project-config change: I8cbc36266eadaf0dbef20e3e83f4874bcfc9ab3a
Change-Id: I2365619de4ba7b43a049771bac90d2deebaa3e9c