openstack/devstack-plugin-ceph - devstack-plugin-ceph - OpenDev: Free Software Needs Free Tools

Similar to what happened some time ago for the stable/queens branch, we
are now seeing issues again on master after there was a new Paste
release being made a couple of weeks ago. So we need to perform
the same workaround of forcing pip to install PasteDeploy over
the distro version on the master branch.
Change-Id: I3e98a08e78182bdf5ec09b30e31b98435ec29456
Closes-Bug: 1803143

This migrates the legacy jobs to zuulv3 native ones.
While migrating the jobs I got the impressions that they have not
bin maintained recently, which is possible since they are not
voting / experimental.
The regex in the py27 job includes cinder.* tests, and the job
used the all-plugin tox env (which has site-packages enabled) so
my guess is that the job used to run cinder in tree tests.
Those tests are now available in the cinder-tempest-plugin repo
instead, which could be added to these jobs if needed.
Following a similar setup in the cinder-tempest-plugin[0] we add
a file `tempest_blacklist.txt` that can by amended to list tests
that are known to fail for a ceph based deployment.
[0] https://review.openstack.org/#/c/531206/
Co-Authored-By: Jens Harbott <j.harbott@x-ion.de>
Change-Id: I27f85991ca947809d46ae2dee57704e779e6dfb6

This is a mechanically generated patch to complete step 1 of moving
the zuul job settings out of project-config and into each project
repository.
Because there will be a separate patch on each branch, the branch
specifiers for branch-specific jobs have been removed.
Because this patch is generated by a script, there may be some
cosmetic changes to the layout of the YAML file(s) as the contents are
normalized.
See the python3-first goal document for details:
https://governance.openstack.org/tc/goals/stein/python3-first.html
Change-Id: I65217d73f55132dbd1ca48ac421cba504543d1cb
Story: #2002586
Task: #24327 

Simply use $KEYSTONE_SERVICE_URI, which is the value also
exported through OS_* variables.
Change-Id: I07296379fa691fed20ed5b1c53b335b2442a6c4e

The functions support only repositories from shaman,
but they can be extended to support stable repositories as well.
Change-Id: I633f80c0f74a69e6d67674398f2013f09b96c661

Clean the "CentOS" branch, so that it should work for all
CentOS/RHEL variants (7.x). Make sure to install the
required repositories also in the non-Manila case.
Fedora already ships Ceph packages so does not need additional
repositories.
Do not try to install ceph-radosgw twice.
Finally, explicitly fail for other distributions (not supported
de-facto even in the current code).
Change-Id: I83b1fffad405bc122bc55ef1173f5239f41d2b36

The remaining occurrences of cat <<... EOF populate non-INI files.
Remove few security checks which are not needed, because iniset
takes care of checking the existence of the section and keys and
adding or replacing them if needed.
Change-Id: I4951e1f2f456b3c6f83fd4868db90fae9d811fbe

The integration with keystone through the PKI tokens (which is removed
anyway since Ocata) and SSL is now disabled by default, and enabled
only if a new variable (CEPH_RGW_KEYSTONE_SSL) is explicitely set
to True.
Change-Id: I4884a8e63c04451e83eb7a104ad7eb7d520b0921

It was removed from DevStack before the Ocata release, see
https://review.openstack.org/#/c/391380/
The support for the only alternative (templated catalog) was
removed even earlier, before Mitaka:
https://review.openstack.org/#/c/278333/
This means that the entry points for Swift can always be
configured when RadosGW is enabled.
Change-Id: I25c1f963665fab1dc77db40a2742dc29a9c1ee0b

The names were used for the same entity, so the default was never
applied. Use the variable defined first (and whose name looks
more consistent with the variable normally used to defined
the identity name).
Also, change the default version to 3. As no one seems to be
using it right now on the gates, it should not break anything.
Change-Id: I4cbb104198e2cdaa8e3e8c4b1f1c20dc3e0048b5

The workaround in order to fix the issue with Paste/PasteDeploy being
pulled in as system packages when installing ceph is only needed for the
Pike UCA. Assume that we only use that when devstack is running at the
stable/queens branch.
Change-Id: I0bbf04be015b777b3feb2c5e837b1f46f81cf3e9

Since manila-tempest-tests is now installed system-wide and not in a
virtual environment, use tempest installed system-wide to run them.
Depends-On: I0844cdb48b9d415bd341c4687620412178be871b
Change-Id: I8abe6ba29766ecaa7faade06a1b23520733139a2
Signed-off-by: Rishabh Dave <ridave@redhat.com>

TEMPEST_SERVICES global variable is not supported
by devstack since long back.
- I380dd20e5ed716a0bdf92aa02c3730359b8136e4
- I9c24705e494689f09a885eb0a640efd50db33fcf
Service availability of tempest known services will be
set by devstack with local check.
- I02be777bf93143d946ccbb8e9eff637bfd1928d4
This commit removes the unused TEMPEST_SERVICES setting
Related-Bug: #1743688
Change-Id: I4ad77632027438537a38d46145b389ef6de06c98

The regex in 84068e was inverted.
Change-Id: Ibac88f2f5a2fbd52a3111d87d48352dd6dd813c4

This assumes that nobody is using this plugin
on outdated versions of Fedora.
Also don't recommend an outdated Fedora.
Change-Id: I858b78ee1a02e98a5bf4545ee315328e3fab0010

Fix in order for the tempest jobs run manila's
tempest plugin from the new plugin repository.
Change-Id: Id07f965e5b73d20ca99e75760d196d55c7b6fbc8

... and export counter in RADOS pool.
Change-Id: I192a0aebdd7cad11c9aa2e8e6b3c4e814f55a12d

We should only attempt to install the python3 packages
if we're running in a python3 environment.
Change-Id: Ic5e3046917a51d884091b449a433259c049ac016
Closes-Bug: #1724249 

Zuul v2 uses 'jenkins' as user, but Zuul v3 uses 'zuul'.
Using $USER solves it for both cases.
Change-Id: I8cddbefaf006b6d54285fffe2a245a4a8ea8b251

The gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial job is
currently failing at a rate of 100% since we updated to use the Pike
UCA. The Ceph packages in the Pike UCA are pulling in python-paste and
python-pastedeploy. When devstack pip installs a newer version of
python-paste to satisfy upper-constraints, we end up with Paste
installed from pypi and python-pastedeploy installed from the Ubuntu
archive and Keystone fails to start with the error:
 "ImportError: cannot import name deploy."
This adds a hack to force an update of PasteDeploy from pypi during the
devstack "stack install" phase to work around the problem and get the
job passing again.
Change-Id: Idf28de8faa56c2d387291c7d14ac6d4bf1c74c6a

Recently, I noticed in the devstack experimental job
gate-tempest-dsvm-py35-full-devstack-plugin-ceph-ubuntu-xenial-nv that
when Glance runs under python 3, it was unable to import rados and rbd
because the Ceph plugin doesn't install the python 3 packages. This
adds the packages to the list for install.
Change-Id: Ib4f956e45885df583624bd7c24989472f44eb365

Change [1] removed the tempest pin from the
pre test hook in manila proper.
Remove it here as well.
Closes-Bug: #1711815
[1] Ia402fb79e1de5c9925f575d635b9928475095f86
Change-Id: I2a6e1854718df272eb40d68cbe41d9ddfb962b71

ceph-mgr daemon is required as part of Ceph cluster from
luminous release onwards.
Change-Id: I0903b3cd9fa84041bcd7ed5034d8a60d14759e92
Closes-Bug: #1708169 

Change-Id: I1d696c63baad4d117e6e7e87242751c2d550b739

The manila cephfs-nfs driver jenkins gate has been failing for a while
since it started using Ceph >= 12.1.1. The manila driver is unable to
fetch Ceph cluster usage stats as it did with earlier Ceph releases.
Temporarily revert to using Kraken 11.2.x stable packages to unblock
the failing gate.
Partial-Bug: #1708169
Change-Id: I4e842e2cee14755e02714e856114a5eabb5613b3

... for cephfs-nfs driver
Change-Id: I7d59ce288a245e7399b38020282be8e7c7fe62f7

This command no longer exists.
Closes-Bug: #1700160
Change-Id: I9feaff1e8d7e6505d36d0702ef531a95289f9618

This is to workaround this test which is not currently
working:
https://review.openstack.org/#/c/471352/
Change-Id: I9506e326f9432eda3a057996805d4b0cd7a91446

for tempest configuration in manila post test hook to "nfs"
so that CIFS tempest tests will be skipped. Its
default value is ["nfs", "cifs"].
Change-Id: I96a5f6bfd5364740689538538bac8bb59187b9b6

... in the cephfs-nfs driver's Jenkins gate.
Change-Id: I8ba601835f8f1d989422c3e047dd84edd9441a3d

- 24 and 25 are the currently supported releases.
- 22 and 23 are now unsupported and are EOL.
Fedora Releases
https://fedoraproject.org/wiki/Releases
Change-Id: Ib03fdb0855eb02f7495b3835d8016cbe08508b1e

... for Manila. Tested to work in Ubuntu Xenial
and CentOS 7 environments.
Co-Authored-By: Tom Barron <tpb@dyncloud.net>
Change-Id: Ib2e806fefdd0548b06ff6aadd4b6a8e62001ea33

... for Ceph versions >= 11.1
For versions of Ceph >= 11.1, starting ceph-mon service doesn't
implicitly call ceph-create-keys to create the client admin key [1].
Call ceph-create-keys after starting MONs to do so.
[1] http://tracker.ceph.com/issues/16036
 cc441cbd6e
Change-Id: Id67cc875ce5e8bca7430465a30e2b02dd81c7e6a

When setting up Manila with CephFS in Ubuntu machines, don't
always install Ceph packages for xenial. Allow installation
of Ceph packages for trusty in Ubuntu trusty machines.
Change-Id: I70d26a3db2034331c8101795b432831718a15273

For manila-CephFS CI, latest Jewel Ceph packages built using
gitbuilder were installed. Jenkins based Ceph build system is going to
replace the gitbuilder based build system. Switch to using packages
built by the newer system.
Change-Id: Ic9b76bc5754dc8b98a11da9fadbeb9d6ace87d5b
Closes-bug: #1642678 

As per,
http://lists.openstack.org/pipermail/openstack-dev/2017-February/112872.html
a devstack plugin's pre_test_hook script can set options
in local.conf and not in localrc. Do so to unblock gate,
gate-manila-tempest-dsvm-cephfs-native
Closes-bug: #1668197
Change-Id: Ic426935f0c5c5aa66fddf496b6b97350ec56b5e5

Ceph doesn't support volume swapping so disable that feature in
tempest.
Change-Id: I8f3da1d730113b100526662ba7ec71f603ab36da

There is no snapshot manage feature on ceph backend and
tests for the same should be skipped on ceph jobs.
There is devstack variable to do so.
Currently gate-tempest-dsvm-full-ceph-plugin-src-glance_store-ubuntu-xenial
is failing and blocking glance-store
Details:
http://lists.openstack.org/pipermail/openstack-dev/2017-January/110257.html
This commit sets TEMPEST_VOLUME_MANAGE_SNAPSHOT to False for ceph jobs.
Change-Id: Ie3eca6db5c76153d46e4dfbe9b552e4025347d31

In [1] the default value for CEPH_LOOPBACK_DISK_SIZE_DEFAULT was changed
to be VOLUME_BACKING_FILE_SIZE instead of a fixed value. This will break
in some cases as there is no fallback defined for
VOLUME_BACKING_FILE_SIZE in devstack/lib/ceph.
[1] https://review.openstack.org/403988
Change-Id: I117e36db442c8ef8c469e4149629e8a27d22da22

This patch adds the team's and repository's badges to the README file.
The motivation behind this is to communicate the project status and
features at first glance.
For more information about this effort, please read this email thread:
http://lists.openstack.org/pipermail/openstack-dev/2016-October/105562.html
To see an example of how this would look like check:
https://gist.github.com/12e6197629898729300789fabbaa9660
Change-Id: Ic349b1f9fecc18b1525c64248471fa91039bb288

The backing disk currently created is 8GB. devstack-gate
sets that to 24GB when running Tempest. We're seeing ceph
job failures due to NoValidHost in the cinder scheduler
because 8GB isn't enough capacity for Tempest runs. So this
change uses the same backing disk size for the ceph jobs as
we get in the default devstack setup, which uses LVM.
Depends-On: I71be308c8373e9ac429b901c374100c6b3c1e59d
Change-Id: I788eefa6c1d427bf51d2d3d40be4abe0336443e7
Closes-Bug: #1645530