openstack/devstack-plugin-ceph - devstack-plugin-ceph - OpenDev: Free Software Needs Free Tools

Change-Id: I293ccf2111e2983465fcd8c12b3ea931ad446344

We should be using "ceph fs set <fs_name> allow_new_snaps true"
instead of using the mds call
Change-Id: I7a709d49ec4c93b2a06e961ca8ca1414cf4c1918
Closes-Bug: #1838790 

ceph-create-keys was being used to create keys.
This command has been deprecated and it does nothing.
For this reason, keys were not available when starting
daemons and this made the deployment to fail.
This change adds the correct key creation using ceph-authtool.
Change-Id: Ie1a41f67bbf1c7a58e89168b0c8ad63f101648b4
Co-Authored-By: Giulio Fidente <gfidente@redhat.com>

Depends-on: https://review.opendev.org/#/c/671134/
Depends-on: https://review.opendev.org/#/c/670510/
Change-Id: Ib812d2b960b1375129c7ff74c63c1acd2649102f

devstack-plugin-ceph is branchless and its jobs run on
stable branches. To tests the devstack-plugin-ceph changes
for stable branches, let' add the stable version of
devstack-plugin-ceph-tempest-py3 job in experimental pipeline.
Adding in experimental pipeline as we do not really have to run
those all the time on check gate but can be run on demand.
Change-Id: I9010f0d6226304537adcbe4e3f97250c754a7de9

Related-Bug: #1653953
Depends-On: https://review.opendev.org/#/c/668119/
Change-Id: I1aa87fc16a37b6051e55c35ae69ef911a87d4c11

Related-Bug: #1653953
Depends-On: https://review.opendev.org/668118/
Change-Id: I132e26486782918d31a9ac10332f2802342883a9

The dependent nova fix resolves the issue with rbd and
shelve/unshelve testing in tempest. Re-enabling the test
needs to be branch-specific since devstack-plugin-ceph
is branch-less but the fixes in nova will be branch
specific so we don't want to enable this on any nova
branch that does not have the fix since it would break
the job on those branches.
Related-Bug: #1653953
Depends-On: https://review.opendev.org/457886/
Co-Authored-By: Matt Riedemann <mriedem.os@gmail.com>
Change-Id: Ifbfc15033020c430a93574f2726b2279c9be130d

This patch adds support for installing, configuring all required
ceph-iscsi related packages and daemons. ceph-iscsi requires kernel
version 4.16 or greater.
To enable set
ENABLE_CEPH_ISCSI=True
Adds new config options
CEPH_ISCSI_TARGET_IQN - the rbd-target-gw iqn to use to export volumes
CEPH_ISCSI_API_HOST - the rbd-target-api ip to use to listen on
CEPH_ISCSI_API_PORT - the rbd-target-api port
CEPH_ISCSI_API_USER - the rbd-target-api user for auth against the api
CEPH_ISCSI_API_PASSWORD - the rbd-target-api password for auth against the api
The following packages are required and their deps
tcmu-runner - this requires kernel version 4.16 and > to work. Ubuntu
doesn't have a prebuilt package for this, so we pull it down and compile it.
rtslib-fb - has a pypi package
configshell-fb - has a pypi package
targetcli-fb - doesn't have a pypi package.
ceph-iscsi - doesn't have a pypi package.
The reference documentaion to setup ceph-iscsi targets and daemons:
http://docs.ceph.com/docs/master/rbd/iscsi-target-cli-manual-install
Change-Id: Idfac4883b9d31e80124b8f2f1cc60dd9ce18c594

The manila CentOS jobs break often, and are not
maintained. Let's move them to experimental
so they don't gate every patch. The other manila
jobs currently provide necessary test coverage
to cephfs-native and cephfs-nfs protocols/drivers.
These jobs also run with python3, and we don't
need to support python3.5 any longer.
Change-Id: Ibb752a1d9aa818dd6862bcbbce4d6b6e366b4800

Allow running api and scenario tests
together in the manila test jobs.
Change-Id: Ifbf998d7c8a27ad22a39f46547d1548c052614d0

Devstack and tempest default to enable ssh validation in
a lot of API tests which can fail intermittently:
 http://status.openstack.org/elastic-recheck/#1808010
Also see bug 1802971.
Since a ceph backend for storage shouldn't really have
any impact on whether or not ssh is going to work in a
tempest run, this change disables it in the
devstack-plugin-ceph-tempest jobs which should make the
job faster and more stable.
Change-Id: Iea948bcc43315286e5c130485728152d4710bfcb

There is really no point in running the tempest.api.network
tests in the devstack-plugin-ceph-tempest job since a ceph
backend shouldn't make any difference to those tests.
This should speed up the job slightly and more importantly
avoid unnecessarily random failures from tests that don't matter
for this setup.
Change-Id: Icf8d776a648b93b232b83fd4493e00b5f76b3ebf

Change-Id: I63c00c728033649be33511a068b52b9f15cfa5d4

This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:
http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html
Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.

TrivialFix
Change-Id: Icd481b55e7684a67175fe6e5c2e2ef62d6e7046c

This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.
This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.
This update should result in no functional change.
For more information see the thread at
 http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html
Change-Id: I0ebfde9478ec07f9f8649e239adfccbf6f7431c6

Below patch enabled the multiattach feature in devstack ceph job
without branch variant. Multiattach feature is only available
from stein.
- https://review.openstack.org/#/c/638499/2
This commit makes two variant of job with multiattach enable from
stein onwards only.
Change-Id: Ib0a5667a6125fa5ae430086db93baf5a53be4877
Closes-Bug: #1820098 

Depends-On: Ie3945427b54544a3b411c23bffdad1acb5e508e1
Change-Id: I520b3db106f72cddabfde4d71eee67607d7a4558

We are running a handful of scenarios that aren't related
to storage here.
Change-Id: I8a22fcfb0454e2267179d07fe67b9e04927145ff

Bring CentOS in line with ubuntu and use luminous rather than
jewel for all manila builds.
Change-Id: Ib0924e42551d25b68ee22ffb9b52caba302ac19c

Tempest change I522a15ba3dbfee5d8ef417e43288a12319abf6ff is
enabling volume multiattach testing in the tempest-full and
tempest-full-py3 job configurations which we inherit for
devstack-plugin-ceph-tempest and devstack-plugin-ceph-tempest-py3
respectively. Since the rbd driver in cinder does not support
multiattach yet we need to disable multiattach testing for these
jobs.
Change-Id: Ia9bd6a42f33a7d9fe3041cf5c751ab691df19cb7

Change-Id: I82cc5232870a820c4c8a0526eb44de45cd67a369

Builds for ubuntu bionic are working now.
We need newer features not yet packaged for ubuntu bionic
for python3 and ceph volume chmod support.
Change-Id: I20fa99c6f6da7d4042b50020ab0e9f34e7ac2de6

Add MANILA_TEMPEST_MIN_API_MICROVERSION and
MANILA_TEMPEST_MAX_API_MICROVERSION to the ci
post_test_hook. These options default to
the values specified in the api_version_request.py
file. On master branch don't set these as we
want to run with the latest.
Related-bug: #1781671
Change-Id: I085ed175dfbe132c80a6a86ab34a281b3c94f5f0

Use python3 compatible tempest test discovery and
install the relevant python3 libraries.
Change-Id: Iabd1952ddaba85316dec5cf60c89122dc3825912

Change-Id: I893ec0c97540e76d368a3cbfd8ebe63a0685a16b

The shaman ceph and ganesha repos for ubuntu bionic are less
stable than those available with the bionic distro itself so
use the latter.
Also refactor the ceph/ganesha repo configuration so that it
is easier to read and maintain, and since bionic does not ship
with knfs installed make the service stop/disable for these
services robust in that case.
Change-Id: If00cc82cfb076c75a11738dcaca372b0cc7a1848

Finding this file via zuul.project only works if
the project is devstack-plugin-ceph, look for it via
zuul.projects which should work for any run of this
job.
Change-Id: I4bc982fad3b7de9582ef30b115c6829980f61047

c6d3b84 intended to turn on
"TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_luks",
but it didn't since that test is marked as slow.
It only turned on
"TestVolumeBootPattern.test_boot_server_from_encrypted_volume_luks".
Change-Id: Ifa8ec1651be95dd6d8d30ed5d04e9ac692209e50

Enable encrypted volume testing.
Change-Id: I96739a3592241aff6326662d15f36b39f1772356

Continue to use the upstream shaman repos for manila so
we can test changes prior to their inclusion in the distro.
Also overwrite /etc/ganesha/ganesha.conf rather than
appending to it since the file now ships with configuration
that breaks ganesha with CephFS FSAL.
Change-Id: I6db16684fcf1e4ad573e25f0475b283cd85dc2c7

Similar to what happened some time ago for the stable/queens branch, we
are now seeing issues again on master after there was a new Paste
release being made a couple of weeks ago. So we need to perform
the same workaround of forcing pip to install PasteDeploy over
the distro version on the master branch.
Change-Id: I3e98a08e78182bdf5ec09b30e31b98435ec29456
Closes-Bug: 1803143

Ubuntu Bionic has Ceph Luminous packages available natively, so we don't
need to set up any additional repositories.
Add a delay after starting ceph-mon in order to allow it to run
properly.
Change-Id: I2c40c4b4e49145d679affaba678f7117ce73a1d1

This migrates the legacy jobs to zuulv3 native ones.
While migrating the jobs I got the impressions that they have not
bin maintained recently, which is possible since they are not
voting / experimental.
The regex in the py27 job includes cinder.* tests, and the job
used the all-plugin tox env (which has site-packages enabled) so
my guess is that the job used to run cinder in tree tests.
Those tests are now available in the cinder-tempest-plugin repo
instead, which could be added to these jobs if needed.
Following a similar setup in the cinder-tempest-plugin[0] we add
a file `tempest_blacklist.txt` that can by amended to list tests
that are known to fail for a ceph based deployment.
[0] https://review.openstack.org/#/c/531206/
Co-Authored-By: Jens Harbott <j.harbott@x-ion.de>
Change-Id: I27f85991ca947809d46ae2dee57704e779e6dfb6

This is a mechanically generated patch to complete step 1 of moving
the zuul job settings out of project-config and into each project
repository.
Because there will be a separate patch on each branch, the branch
specifiers for branch-specific jobs have been removed.
Because this patch is generated by a script, there may be some
cosmetic changes to the layout of the YAML file(s) as the contents are
normalized.
See the python3-first goal document for details:
https://governance.openstack.org/tc/goals/stein/python3-first.html
Change-Id: I65217d73f55132dbd1ca48ac421cba504543d1cb
Story: #2002586
Task: #24327 

Simply use $KEYSTONE_SERVICE_URI, which is the value also
exported through OS_* variables.
Change-Id: I07296379fa691fed20ed5b1c53b335b2442a6c4e

The functions support only repositories from shaman,
but they can be extended to support stable repositories as well.
Change-Id: I633f80c0f74a69e6d67674398f2013f09b96c661

Clean the "CentOS" branch, so that it should work for all
CentOS/RHEL variants (7.x). Make sure to install the
required repositories also in the non-Manila case.
Fedora already ships Ceph packages so does not need additional
repositories.
Do not try to install ceph-radosgw twice.
Finally, explicitly fail for other distributions (not supported
de-facto even in the current code).
Change-Id: I83b1fffad405bc122bc55ef1173f5239f41d2b36