openstack/devstack-plugin-ceph - devstack-plugin-ceph - OpenDev: Free Software Needs Free Tools

This job covers the popular usecase of not using rbd for ephemeral
storage for nova while it is used for Glance and Cinder.
This job additionally enables the recently implemented direct download
of rbd hosted Glance images into the file based ephemeral store of Nova.
Change-Id: Ieddd803fccb097657fffe30ef87c1c8385e7eb28

Per the best practices.
Change-Id: I7896886d356a75a896660e08e2152c69534fa26f

We seem to have been requesting the latest build from the Shaman API
provided by Ceph however we did not filter the status to be 'ready'
which meant that we sometimes got 'building' status repositories that
cause the jobs to fail.
This should stabilize jobs a lot more by using only the ready
repositories.
Change-Id: Iff47bbe4954da3ab582a007ba04b3df50ae21cd4

We are using iSCSI protocol to operate the volume, instead of ceph
in that case. The storage protocol ceph is used as Ceph RBD.
Change-Id: I288b08279311991a79fdad25e43a51366de7876d
Signed-off-by: Kevin Zhao <kevin.zhao@linaro.org>

It looks like the default bluestore that came with nautilus
is limiting us to a for-testing size of 10G regardless of what the
actual ceph volume is supposed to be. Try setting it to the size
we want (minus some room for overhead).
Closes-Bug: #1888895
Change-Id: I4e5233df826af6eeddc4e77820958909d4fe7d45

Added mgr "allow *" for manila user
Change-Id: I4a0dfbe4925554adbb40fe3d24e44098589f1a09

- manila-tempest-minimal-dsvm-cephfs-native and
 manila-tempest-minimal-dsvm-cephfs-nfs are both
 legacy jobs that use the now deprecated
 devstack-gate.
- We no longer need the CentOS based experimental
 jobs for manila since the manila team plans to
 delete them from their tree. Newer CentOS experiments
 can happen with the native zuulv3 style jobs.
- The jobs added are inheriting manila-tempest-plugin jobs,
 but currently don't override anything - there's a plan
 to supply overrides soon to accommodate the ceph support
 matrix that the manila team wants to test via this
 repository.
- Also make minor refactors to existing base
 jobs to make reuse of certain things easier
A future commit will remove the pre-test-hook and
post-test-hook scripts; they're presently there to allow
testing legacy jobs against older releases of
openstack/manila.
Change-Id: I20ab47cf7846e7c0291ef4771cc93fd85d0452b0
Signed-off-by: Goutham Pacha Ravi <gouthampravi@gmail.com>

Limit ceph iscsi ubuntu support only to Focal. As only Focal has the
required packages and we don't want to install ceph-iscsi from master
source code which is changing all the way is not a good way for CI.
Remove api_host setting, let it be ceph iscsi default setting "::"[1],
Which references to ceph-ansible configuration[2]. This makes
rbd-target-api accessible for both localhost(gwcli) and HOST_IP.
Otherwise iscsi gateway creation will fail in newer version
ceph-iscsi[3].
Change minimum-gateways default to 1 for newer version ceph-iscsi,
otherwise client creation will fail on all-in-one setup[4].
Fix gateway creation failure with error:
"The first gateway defined must be the local machine"
Ceph-iscsi requires gateway name to be FQDN, sometimes hostname is not
the same as FQDN.
Fix stop_ceph_iscsi run twice.
Remove setup_packages_for_iscsi as openstack won't support projects that
require packages installed with git clone.
[1]: https://github.com/ceph/ceph-iscsi/blob/master/ceph_iscsi_config/gateway_setting.py#L174
[2]: https://github.com/ceph/ceph-ansible/blob/master/roles/ceph-iscsi-gw/templates/iscsi-gateway.cfg.j2
[3]: https://github.com/ceph/ceph-iscsi/issues/193
[4]: https://github.com/ceph/ceph-iscsi/issues/195
Change-Id: I215b870c66cc27f10f8a6bf18d93feda477a3970

CephFS drivers don't support cloning snapshots
into shares. Manila's tempest tests need to be
told this explicitly. So, update the gate hooks
that control tempest configuration.
This affects all legacy style CephFS jobs that
use devstack-gate. These jobs will go away soon,
and be replaced with Zuulv3 native style job
definitions and this patch is a stop-gap,
because devstack-plugin-ceph is a branchless
devstack plugin, and we need to support legacy
style jobs for a while.
Change-Id: I885ff78586540d01590f98cca2a00d55161dbe04
Signed-off-by: Goutham Pacha Ravi <gouthampravi@gmail.com>

The use of apt to refresh repos is discouraged and prints the following
warning when used:
WARNING: apt does not have a stable CLI interface. Use with caution in scripts
The use of apt also appears to break on our Xenial based branches where
we enable the Xenial Queens Ubuntu Cloud Archive repos that are
unsigned. When using apt to refresh these repos apt returns a failure,
while apt-get only returns a warning.
This change simply switches to apt-get to avoid the above warning and to
allow the older Xenial based stable branches to continue to work.
Change-Id: Ic534a47fff17d36eb1ef128ad6af70a34ed942c9

Manila doesn't provide snapshots unless you
assert that you want them via share type
extra-specs. Tests that require "snapshot_support"
do this by creating a share type with that
extra-spec; so we don't need this config
option to be set.
It's actually detrimental when wanting to enable
snapshots on the Ceph driver [1]
[1] https://review.opendev.org/736827/
Change-Id: I41d6aa885e22d3f21150d49ae7e4a8f369cc96f5
Signed-off-by: Goutham Pacha Ravi <gouthampravi@gmail.com>

We were using Jewel/Luminous. Being a developer tool, we should
be sticking with newer versions.
This patch set also adds the following changes
- Refactor setup_packages_for_manila_on_<distro> functions
- Removes ceph_init call. Seems it lost relevance
- Removes APT repositories setup. We directly use shaman/chacra
packages directly
- Removes RUN_AS env var. It has been a while since Ceph
defaults to ceph user to run all the services. No longer needed.
- Removes sleep calls for services start checks. We use systemctl
checks.
- Adds more debugging logs
- Uses the search API from ceph shaman instead of hardwiring urls
- Adds the jq (JSON processor) dependency
Change-Id: I51c90e592070b99422e692d5e9e97083d93146e8

Change-Id: Ieb04d25e0ce0b061e4e2ac26b36e371bd666eedd
Signed-off-by: Kevin Zhao <kevin.zhao@linaro.org>

Switch to openstackdocstheme 2.2.1 and reno 3.1.0 versions. Using
these versions will allow especially:
* Linking from HTML to PDF document
* Allow parallel building of documents
* Fix some rendering problems
Update Sphinx version as well.
Set openstackdocs_auto_version to not auto-version the documents.
Set openstackdocs_auto_name to use 'project' as name.
openstackdocstheme renames some variables, so follow the renames
before the next release removes them. A couple of variables are also
not needed anymore, remove them.
Change pygments_style to 'native' since old theme version always used
'native' and the theme now respects the setting and using 'sphinx' can
lead to some strange rendering.
See also
http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014971.html
Change-Id: I71e78d4ff18b0fd92f16f0917283eddad543250f

Add ussuir and train stable job and remove the rocky
and queens which are EM.
Change-Id: I47406690552fc6f8d6fcc1cc04d94cf285d42037

Implement an old TODO item, but in the other way: instead of
enabling the feature on certain branches, disable it on the old
branches (anything still open - even EM - before queens)
and rely on the default value (True) otherwise.
Change-Id: Id5dedf496b355362031deefab1573d052fc72a66

IPv6 export policy support was added to the Manila
CephFS-via-NFS driver in the Train release. This
devstack plugin is branchless, so we'll need to
check for the relevant branch to setup the ipv6
test harness.
The pre-test and post-test hooks will be deleted soon
when devstack-gate is dropped by manila test jobs
in the Victoria cycle. So this code is temporary
but required to make ceph-via-nfs tests pass
in stable/queens-stein branches.
Change-Id: I91a54a8f72d1d514afbfa769c7e8f2ac21d502dc
Signed-off-by: Goutham Pacha Ravi <gouthampravi@gmail.com>

QA people pointed out that It should be set here, not in the job.
Change-Id: Id925fe667472f7a71ac236aec136dd7b59e43794

The Manila driver is not WIP, it's used in many
production deployments since the Liberty release.
Set the option ENABLE_CEPH_MANILA to True in case,
so that the manila related setup will occur automatically
if manila is enabled in devstack.
Change-Id: Iec66b201fd775125505fd229963f2d0a5e64c8cb
Signed-off-by: Goutham Pacha Ravi <gouthampravi@gmail.com>

This reverts commit c1fc88670c.
Depends-On: https://review.opendev.org/#/c/725812/
Change-Id: Ic58b68d431a7226fdd16b5ac2259aff92c9ecd16

Nova implemented the feature for rescuing the BFV instances.
- https://specs.openstack.org/openstack/nova-specs/specs/ussuri/approved/virt-bfv-instance-rescue.html
It is not working with ceph backend as new tests added in Tempest are failing ceph job
Related-Bug: #1876330
Change-Id: Ie675610a040e9f4c164b51b6b61b2a8094ecd09f

In the master (Victoria cycle), manila's
devstack plugin will handle:
- enabling NFS ports for access from guest VMs
- setting up bgp speaker and peer for routing to
 ipv6 tenant networks
I'll clean this up up entirely when we backport
these changes to older branches of manila.
Depends-On: https://review.opendev.org/#/c/724202/
Depends-On: https://review.opendev.org/#/c/724204/
Change-Id: I1087c562ff753cb764a174130ce1fed21a9c7b12
Signed-off-by: Goutham Pacha Ravi <gouthampravi@gmail.com>

Add an extra check to verify that ceph-mon service has started
before trying to retrieve its version.
An already existing function is being used, wait_for_daemon
Since we now guarantee that we are using systemctl, we can use
the is-active feature. If the service is running, the exit code
will be 0.
Also, change the command used to get the mon version. We can use
the ceph version command provided by the new cli instead of using
ceph daemon command which requires more privileges.
Change-Id: I2d4f23721107c4b5cb1c09719c55424eecd0c880
Closes-Bug: #1872855 

Upstart is not being used anymore (last release was Sept 2014)
Doesn't make sense to keep bits for this
Make this plugin compatible only with distros with systemctl
Change-Id: I4fd533985d5bce3e0513c46558763629fb1539fc
Closes-Bug: #1874457 

This patch updates/adds the contributor documentation to follow
the guidelines of the Ussuri cycle community goal[1].
[1] https://governance.openstack.org/tc/goals/selected/ussuri/project-ptl-and-contrib-docs.html
Story: #2007236
Task: #38554
Change-Id: I1b544695dc5cc1c9f2cdd5fc3f686dae94d52aaa

devstack-plugins-ceph does not have the doc infra set
up and no doc publish job.
This is need to add the contributor doc as per the ussuri
community goal.
Change-Id: Ib62c34b52b28b753f7f3046d51492fd8d9fb6193

While the job itself is not currently voting within openstack/nova the
use of the plugin essentially is within the nova-live-migration job.
As such we should really enable voting of this job within this project
with the eventual goal of also enabling it within openstack/nova and
other projects using it.
Change-Id: Id243b92e800e2d6e8b2228fadbd1c72b194bbfb9

devstack-plugin-ceph-tempest is py2 job and not supposed
to run on master gate. devstack-plugin-ceph master gate already
run its py3 version devstack-plugin-ceph-tempest-py3.
We need to keep the devstack-plugin-ceph-tempest definition
for stable branches gate.
Change-Id: I6c512b29a0346db1f92de34b1dbfb56958fa2cad

I5698e451861828a8b1240d046d1610d8d37ca5a2 introduced support for this
test in Stein so enable the compute feature from Stein onwards.
Blueprint: extend-in-use-rbd-volumes
Change-Id: I6b2575b12fdb9636f106b3ca28681e29ad3eb8ac

In change I43de36ab570af40bc837d859568f5428c167f083, we started
installing the 'python3-' variants of the 'python-' packages installed
by the 'ceph-common' package in 'install_ceph_remote'. In that change,
we made the assertion that we didn't need to "both hiding this behind
the 'python3_enabled' function or 'USE_PYTHON3' variable since (a) these
packages can happily coexist, (b) projects should be switching to Python
3 now, and (c) using these would require sourcing additional, otherwise
unnecessary devstack libraries just to use this simple function".
However, this didn't take into account the fact that this plugin is not
versioned. This means *any* job that uses this plugin, including things
like stable/pike, will now attempt to install the 'python3-' packages.
Since the base OS on stable/pike, Xenial (16.04), doesn't actually
provide these 'python3-' variants, the command will always fail.
Start hiding the command behind a 'use_python3' function call. Hopefully
by time we drop Python 2 support here we won't have maintained branches
using Xenial. If we do, we'll need to branch or something like that.
Change-Id: I890a1e673fde212218eff4fcf82f7941b3283e5f
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Closes-Bug: #1853280 

Previously, this command would simply install the 'ceph-common' package,
which includes a number of Python dependencies. Unfortunately, these
packages are the 'python-' variants [1] and there doesn't appear to be a
'ceph-common-python3' package or similar to install the 'python3-'
variants. As a result, we need to install the 'python3-' variants
manually to ensure consumers of the plugin (like nova) can switch to
Python 3 in their tests.
We don't bother hiding this behind the 'python3_enabled' function or
'USE_PYTHON3' variable since (a) these packages can happily coexist, (b)
projects should be switching to Python 3 now, and (c) using these would
require sourcing additional, otherwise unnecessary devstack libraries
just to use this simple function.
[1] https://packages.ubuntu.com/bionic/ceph-common
Change-Id: I43de36ab570af40bc837d859568f5428c167f083
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

Cleanup of CephFS data and metadata pools for manila was only
being done for external CephFS installs.
Do it whenever manila is enabled.
Change-Id: Iee238f91b04231d1e41774a49615023fe26e56f3

Related-Bug: #1653953
Depends-On: https://review.opendev.org/#/c/668123/
Change-Id: I3e1f9683a28a2e2e67a0327031f0670f33391126

Use devstack's vercmp function instead of
bc for version comparisons.
Change-Id: I6436a6f836ad1ab9594858986bfdb8d6d6dbc392