openstack/devstack-plugin-ceph - devstack-plugin-ceph - OpenDev: Free Software Needs Free Tools

Update the list of tested stable branches (excluding stein which is EM).
The nodesets for the stable branches are now pinned because
their testing environment are well defined and won't change.
Also, remove the -py3 suffix: those jobs are all -py3 based
(the suffix was mostly requires in the transitional time
between py2 and py3).
Change-Id: I48f8574960df51fc47f80fa17091fd13ec71aa99

Having debug_ms enabled populates logs with too much
information. This makes it harder to debug in most cases.
Disable by default.
Change-Id: If180f9be42e95492ab457420e48a494fd2689e6e

As with tempest.api.network the identity tests have nothing to do with
ceph so lets blacklist them and hopefully reduce runtime.
Change-Id: Ibcda268021b236bc049e21b7ddd9e99a7c24dace

Add a glob pattern at the end of the file removals so that we
also capture items like
 nfs-ganesha-ubuntu-libntirpc-3_0-focal.list.save
which can otherwise cause issues next time we stack
(install_prereqs sees it and gets confused).
Change-Id: I6bb75a75972c47fa87f0d703dda242d2b9bb07b7

When working locally it is nice to be able to
run unstack.sh and not have too much extra stuff
to clean up by hand before running stack.sh again.
So clean up CEPH_DATA_DIR, contenta and mounts.
Also fix a spot where '/var/lib/ceph' (the default
value for CEPH_DATA_DIR) was hard-coded.
Change-Id: I43ff06ca38f42ea4305b2c862902bf6f612e32f0

To use the snapshots capability that the cephfs drivers have
we need to set the share type extra spec enable_snapshot support
to True
Change-Id: I45dddd49578eaca81276bbf44a44701c80b8a8c4

We are using NFS Ganesha 2.7 which is not longer supported
by the Ganesha community. Let's use v3.0 instead.
Closes-Bug: #1898952
Change-Id: I69543fd33c98817bba76c6712ae91264c532c12c

Disable caching done by ganesha when ganesha
is used as CephFS's gateway (manila's usecase)
This is the recommended Ganesha setting as libcephfs
already caches information. Doing so helps avoid
cache incoherency issues.
Change-Id: Iacd575ac61b39ab9dcf8814514b5e3267570a17f
Closes-Bug: #1898950 

I'm not sure why this isn't failing in the gate, but locally I get
100% fail if I don't add the release key before the apt-add-
repository. This seems to be a change in focal as it worked fine
for me before upgrade.
This moves the key fetch/add before the repo.
Change-Id: I50d840986d630af033a7a3e47bb1e9c52e889791

This patch updates the override-defaults to support
setting the CINDER_DRIVER, CINDER_ENABLED_BACKENDS
and CINDER_ISCSI_HELPER env vars for devstack to
ceph_iscsi if and only if ENABLE_CEPH_ISCSI is True.
This is needed by devstack lib/cinder to automatically
configure and enable ceph_iscsi as a backend.
Depends-On: https://review.opendev.org/#/c/662829/
Change-Id: I3447554343edfb5ec84ea65a521168e6d2831dfa

Change-Id: Ic301602c7aada2b64d119b829b6919ee93a5870f

Remove a limitation from the README that's
not relevant any more and stop rendering
the search page in the PDF.
Change-Id: I4c3409c9a6384adf0c20ab08214b866063d4dea5
Signed-off-by: Goutham Pacha Ravi <gouthampravi@gmail.com>

Given the issues introduced by I0eab8ad25112511b0c4dd4b7796b4cda821bca8b
it would be nice to have a simple RPM based job somewhere in the check
queue to avoid a repeat in the future.
Change-Id: I601b77f2da0574d38ccef7d05db4933f6fdae998

This patch enables building .pdf documentation using new 'pdf-docs'
tox environment. The newly created tox environment creates latex
source codes using sphinx-build and then builds the pdf
documentation using make.
Also, convert README.md to README.rst for more seamless integration
of the README file in the pdf and html documentation.
Story: 2006070
Task: 35457
Change-Id: I5c022478c9be0067c125efb4032c16cf0978330f

This isn't required with dnf and can overwrite package downgrades
working around other issues in the underlying OS.
Change-Id: I6940f06c14a0e41d63d632724e85bb790229f307

As Fedora 32 has dropped py2 support we need to switch over to the py3
based el8 RPMs now. Fedora 31 can support both allowing us to switch
while still technically supporting it.
Change-Id: I79f61d61ccb044b6c24b5747b991045edf25679e

I0eab8ad25112511b0c4dd4b7796b4cda821bca8b incorrectly attempted to write
directly to the /etc/repos.d/ directory as the stack user. This isn't
possible even with sudo so use the config-manager command to import
locally written files as the stack user.
Additionally ensure there is no whitespace at the start of each line
within the files.
Change-Id: I2fbc3b7c7df90328954eb4dcee921bd2ae9b2706

Shaman packages are not stable and generates
breakages in our gates quite frequently.
Use packages available in download.ceph.com instead.
Change-Id: I0eab8ad25112511b0c4dd4b7796b4cda821bca8b

This job covers the popular usecase of not using rbd for ephemeral
storage for nova while it is used for Glance and Cinder.
This job additionally enables the recently implemented direct download
of rbd hosted Glance images into the file based ephemeral store of Nova.
Change-Id: Ieddd803fccb097657fffe30ef87c1c8385e7eb28

Per the best practices.
Change-Id: I7896886d356a75a896660e08e2152c69534fa26f

We seem to have been requesting the latest build from the Shaman API
provided by Ceph however we did not filter the status to be 'ready'
which meant that we sometimes got 'building' status repositories that
cause the jobs to fail.
This should stabilize jobs a lot more by using only the ready
repositories.
Change-Id: Iff47bbe4954da3ab582a007ba04b3df50ae21cd4

We are using iSCSI protocol to operate the volume, instead of ceph
in that case. The storage protocol ceph is used as Ceph RBD.
Change-Id: I288b08279311991a79fdad25e43a51366de7876d
Signed-off-by: Kevin Zhao <kevin.zhao@linaro.org>

It looks like the default bluestore that came with nautilus
is limiting us to a for-testing size of 10G regardless of what the
actual ceph volume is supposed to be. Try setting it to the size
we want (minus some room for overhead).
Closes-Bug: #1888895
Change-Id: I4e5233df826af6eeddc4e77820958909d4fe7d45

Added mgr "allow *" for manila user
Change-Id: I4a0dfbe4925554adbb40fe3d24e44098589f1a09

- manila-tempest-minimal-dsvm-cephfs-native and
 manila-tempest-minimal-dsvm-cephfs-nfs are both
 legacy jobs that use the now deprecated
 devstack-gate.
- We no longer need the CentOS based experimental
 jobs for manila since the manila team plans to
 delete them from their tree. Newer CentOS experiments
 can happen with the native zuulv3 style jobs.
- The jobs added are inheriting manila-tempest-plugin jobs,
 but currently don't override anything - there's a plan
 to supply overrides soon to accommodate the ceph support
 matrix that the manila team wants to test via this
 repository.
- Also make minor refactors to existing base
 jobs to make reuse of certain things easier
A future commit will remove the pre-test-hook and
post-test-hook scripts; they're presently there to allow
testing legacy jobs against older releases of
openstack/manila.
Change-Id: I20ab47cf7846e7c0291ef4771cc93fd85d0452b0
Signed-off-by: Goutham Pacha Ravi <gouthampravi@gmail.com>

Limit ceph iscsi ubuntu support only to Focal. As only Focal has the
required packages and we don't want to install ceph-iscsi from master
source code which is changing all the way is not a good way for CI.
Remove api_host setting, let it be ceph iscsi default setting "::"[1],
Which references to ceph-ansible configuration[2]. This makes
rbd-target-api accessible for both localhost(gwcli) and HOST_IP.
Otherwise iscsi gateway creation will fail in newer version
ceph-iscsi[3].
Change minimum-gateways default to 1 for newer version ceph-iscsi,
otherwise client creation will fail on all-in-one setup[4].
Fix gateway creation failure with error:
"The first gateway defined must be the local machine"
Ceph-iscsi requires gateway name to be FQDN, sometimes hostname is not
the same as FQDN.
Fix stop_ceph_iscsi run twice.
Remove setup_packages_for_iscsi as openstack won't support projects that
require packages installed with git clone.
[1]: https://github.com/ceph/ceph-iscsi/blob/master/ceph_iscsi_config/gateway_setting.py#L174
[2]: https://github.com/ceph/ceph-ansible/blob/master/roles/ceph-iscsi-gw/templates/iscsi-gateway.cfg.j2
[3]: https://github.com/ceph/ceph-iscsi/issues/193
[4]: https://github.com/ceph/ceph-iscsi/issues/195
Change-Id: I215b870c66cc27f10f8a6bf18d93feda477a3970

CephFS drivers don't support cloning snapshots
into shares. Manila's tempest tests need to be
told this explicitly. So, update the gate hooks
that control tempest configuration.
This affects all legacy style CephFS jobs that
use devstack-gate. These jobs will go away soon,
and be replaced with Zuulv3 native style job
definitions and this patch is a stop-gap,
because devstack-plugin-ceph is a branchless
devstack plugin, and we need to support legacy
style jobs for a while.
Change-Id: I885ff78586540d01590f98cca2a00d55161dbe04
Signed-off-by: Goutham Pacha Ravi <gouthampravi@gmail.com>

The use of apt to refresh repos is discouraged and prints the following
warning when used:
WARNING: apt does not have a stable CLI interface. Use with caution in scripts
The use of apt also appears to break on our Xenial based branches where
we enable the Xenial Queens Ubuntu Cloud Archive repos that are
unsigned. When using apt to refresh these repos apt returns a failure,
while apt-get only returns a warning.
This change simply switches to apt-get to avoid the above warning and to
allow the older Xenial based stable branches to continue to work.
Change-Id: Ic534a47fff17d36eb1ef128ad6af70a34ed942c9

Manila doesn't provide snapshots unless you
assert that you want them via share type
extra-specs. Tests that require "snapshot_support"
do this by creating a share type with that
extra-spec; so we don't need this config
option to be set.
It's actually detrimental when wanting to enable
snapshots on the Ceph driver [1]
[1] https://review.opendev.org/736827/
Change-Id: I41d6aa885e22d3f21150d49ae7e4a8f369cc96f5
Signed-off-by: Goutham Pacha Ravi <gouthampravi@gmail.com>

We were using Jewel/Luminous. Being a developer tool, we should
be sticking with newer versions.
This patch set also adds the following changes
- Refactor setup_packages_for_manila_on_<distro> functions
- Removes ceph_init call. Seems it lost relevance
- Removes APT repositories setup. We directly use shaman/chacra
packages directly
- Removes RUN_AS env var. It has been a while since Ceph
defaults to ceph user to run all the services. No longer needed.
- Removes sleep calls for services start checks. We use systemctl
checks.
- Adds more debugging logs
- Uses the search API from ceph shaman instead of hardwiring urls
- Adds the jq (JSON processor) dependency
Change-Id: I51c90e592070b99422e692d5e9e97083d93146e8

Change-Id: Ieb04d25e0ce0b061e4e2ac26b36e371bd666eedd
Signed-off-by: Kevin Zhao <kevin.zhao@linaro.org>

Switch to openstackdocstheme 2.2.1 and reno 3.1.0 versions. Using
these versions will allow especially:
* Linking from HTML to PDF document
* Allow parallel building of documents
* Fix some rendering problems
Update Sphinx version as well.
Set openstackdocs_auto_version to not auto-version the documents.
Set openstackdocs_auto_name to use 'project' as name.
openstackdocstheme renames some variables, so follow the renames
before the next release removes them. A couple of variables are also
not needed anymore, remove them.
Change pygments_style to 'native' since old theme version always used
'native' and the theme now respects the setting and using 'sphinx' can
lead to some strange rendering.
See also
http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014971.html
Change-Id: I71e78d4ff18b0fd92f16f0917283eddad543250f

Add ussuir and train stable job and remove the rocky
and queens which are EM.
Change-Id: I47406690552fc6f8d6fcc1cc04d94cf285d42037