openstack/devstack-plugin-ceph - devstack-plugin-ceph - OpenDev: Free Software Needs Free Tools

Recently, I noticed in the devstack experimental job
gate-tempest-dsvm-py35-full-devstack-plugin-ceph-ubuntu-xenial-nv that
when Glance runs under python 3, it was unable to import rados and rbd
because the Ceph plugin doesn't install the python 3 packages. This
adds the packages to the list for install.
Change-Id: Ib4f956e45885df583624bd7c24989472f44eb365

Change [1] removed the tempest pin from the
pre test hook in manila proper.
Remove it here as well.
Closes-Bug: #1711815
[1] Ia402fb79e1de5c9925f575d635b9928475095f86
Change-Id: I2a6e1854718df272eb40d68cbe41d9ddfb962b71

ceph-mgr daemon is required as part of Ceph cluster from
luminous release onwards.
Change-Id: I0903b3cd9fa84041bcd7ed5034d8a60d14759e92
Closes-Bug: #1708169 

Change-Id: I1d696c63baad4d117e6e7e87242751c2d550b739

The manila cephfs-nfs driver jenkins gate has been failing for a while
since it started using Ceph >= 12.1.1. The manila driver is unable to
fetch Ceph cluster usage stats as it did with earlier Ceph releases.
Temporarily revert to using Kraken 11.2.x stable packages to unblock
the failing gate.
Partial-Bug: #1708169
Change-Id: I4e842e2cee14755e02714e856114a5eabb5613b3

... for cephfs-nfs driver
Change-Id: I7d59ce288a245e7399b38020282be8e7c7fe62f7

This command no longer exists.
Closes-Bug: #1700160
Change-Id: I9feaff1e8d7e6505d36d0702ef531a95289f9618

This is to workaround this test which is not currently
working:
https://review.openstack.org/#/c/471352/
Change-Id: I9506e326f9432eda3a057996805d4b0cd7a91446

for tempest configuration in manila post test hook to "nfs"
so that CIFS tempest tests will be skipped. Its
default value is ["nfs", "cifs"].
Change-Id: I96a5f6bfd5364740689538538bac8bb59187b9b6

... in the cephfs-nfs driver's Jenkins gate.
Change-Id: I8ba601835f8f1d989422c3e047dd84edd9441a3d

- 24 and 25 are the currently supported releases.
- 22 and 23 are now unsupported and are EOL.
Fedora Releases
https://fedoraproject.org/wiki/Releases
Change-Id: Ib03fdb0855eb02f7495b3835d8016cbe08508b1e

... for Manila. Tested to work in Ubuntu Xenial
and CentOS 7 environments.
Co-Authored-By: Tom Barron <tpb@dyncloud.net>
Change-Id: Ib2e806fefdd0548b06ff6aadd4b6a8e62001ea33

... for Ceph versions >= 11.1
For versions of Ceph >= 11.1, starting ceph-mon service doesn't
implicitly call ceph-create-keys to create the client admin key [1].
Call ceph-create-keys after starting MONs to do so.
[1] http://tracker.ceph.com/issues/16036
 cc441cbd6e
Change-Id: Id67cc875ce5e8bca7430465a30e2b02dd81c7e6a

When setting up Manila with CephFS in Ubuntu machines, don't
always install Ceph packages for xenial. Allow installation
of Ceph packages for trusty in Ubuntu trusty machines.
Change-Id: I70d26a3db2034331c8101795b432831718a15273

For manila-CephFS CI, latest Jewel Ceph packages built using
gitbuilder were installed. Jenkins based Ceph build system is going to
replace the gitbuilder based build system. Switch to using packages
built by the newer system.
Change-Id: Ic9b76bc5754dc8b98a11da9fadbeb9d6ace87d5b
Closes-bug: #1642678 

As per,
http://lists.openstack.org/pipermail/openstack-dev/2017-February/112872.html
a devstack plugin's pre_test_hook script can set options
in local.conf and not in localrc. Do so to unblock gate,
gate-manila-tempest-dsvm-cephfs-native
Closes-bug: #1668197
Change-Id: Ic426935f0c5c5aa66fddf496b6b97350ec56b5e5

Ceph doesn't support volume swapping so disable that feature in
tempest.
Change-Id: I8f3da1d730113b100526662ba7ec71f603ab36da

There is no snapshot manage feature on ceph backend and
tests for the same should be skipped on ceph jobs.
There is devstack variable to do so.
Currently gate-tempest-dsvm-full-ceph-plugin-src-glance_store-ubuntu-xenial
is failing and blocking glance-store
Details:
http://lists.openstack.org/pipermail/openstack-dev/2017-January/110257.html
This commit sets TEMPEST_VOLUME_MANAGE_SNAPSHOT to False for ceph jobs.
Change-Id: Ie3eca6db5c76153d46e4dfbe9b552e4025347d31

In [1] the default value for CEPH_LOOPBACK_DISK_SIZE_DEFAULT was changed
to be VOLUME_BACKING_FILE_SIZE instead of a fixed value. This will break
in some cases as there is no fallback defined for
VOLUME_BACKING_FILE_SIZE in devstack/lib/ceph.
[1] https://review.openstack.org/403988
Change-Id: I117e36db442c8ef8c469e4149629e8a27d22da22

This patch adds the team's and repository's badges to the README file.
The motivation behind this is to communicate the project status and
features at first glance.
For more information about this effort, please read this email thread:
http://lists.openstack.org/pipermail/openstack-dev/2016-October/105562.html
To see an example of how this would look like check:
https://gist.github.com/12e6197629898729300789fabbaa9660
Change-Id: Ic349b1f9fecc18b1525c64248471fa91039bb288

The backing disk currently created is 8GB. devstack-gate
sets that to 24GB when running Tempest. We're seeing ceph
job failures due to NoValidHost in the cinder scheduler
because 8GB isn't enough capacity for Tempest runs. So this
change uses the same backing disk size for the ceph jobs as
we get in the default devstack setup, which uses LVM.
Depends-On: I71be308c8373e9ac429b901c374100c6b3c1e59d
Change-Id: I788eefa6c1d427bf51d2d3d40be4abe0336443e7
Closes-Bug: #1645530 

Add config file argument to ceph commands in cleanup_ceph_remote
function so that it will cleanup properly if CEPH_CONF_FILE has
a non-default value.
Change-Id: I0c6dc3494bcf700364af0c578cc9a60b082427d5
Closes-Bug: #1643513 

This commit introduces the support of deploying a containerized ceph
cluster. It relies on the ceph/demo container image available as part of
the ceph-docker project (https://github.com/ceph/ceph-docker).
To enable this scenario, just set CEPH_CONTAINERIZED to True, all the
previous available options will work. So settings like
CEPH_LOOPBACK_DISK_SIZE continue to work.
Change-Id: Ie10155140448c04b88b9920381b54505f7359585
Signed-off-by: Sébastien Han <seb@redhat.com>

Xenial currently ships with Ceph 10.2.2 which has a known issue that
can result in updated images failing to be refreshed when the cluster
is overloaded and drops update notifications [1]. This is most likely
the cause for recent Ceph gate failures [2].
Disabling the new Jewel+ default image feature of exclusive-lock
(revert to Hammer default behavior) should be a workaround for this
issue.
The RBD default features can be overridden via CEPH_RBD_DEFAULT_FEATURES
[1] http://tracker.ceph.com/issues/16404
[2] https://bugs.launchpad.net/cinder/+bug/1627220
Change-Id: Id7fab306e918ba7566ebe2a765d4345fea6c5653
Closes-Bug: #1627878
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

This patch introduce new option GLANCE_RGW_BACKEND, if it set to
true and radosgw enabled, glance is configured with swift backend
options.
Also create glance-swift user with ResellerAdmin role to interact
with radosgw.
CEPH_RGW_KEYSTONE_API_VERSION option was added to specify which
keystone API versioni radosgw will use, default v2.0.
Change-Id: I2cacf511e48720896aa621201d6188bd1d370b6e

Since Ceph supports more architectures such as Aarch64.
Change-Id: I3cb05c4a3eb929e9648ecc05c4f0ba042f9e86ad
Signed-off-by: Kevin Zhao <kevin.zhao@linaro.org>

Signed-off-by: Kevin Zhao <kevin.zhao@linaro.org>
Change-Id: Ifc75a97fa7795647494ab92e94b4dec49064eb87

we remove the download.ceph.com repo during cleanup.
Change-Id: Ia2a62a6307b61623b72dbd7ee391d11cbd27f660
Signed-off-by: Sébastien Han <seb@redhat.com>

http://ceph.com/debian-hammer redirects to
http://download.ceph.com/debian-hammer/
However while setting: CEPH_RELEASE=jewel http://ceph.com/debian-jewel
does not redirect to http://download.ceph.com/debian-jewel and stack
fails. So fixing the repo url by directly using
http://download.ceph.com
Change-Id: Ib8e80e43674c6d8ffd01bc6af56e66e9e254213b
Signed-off-by: Sébastien Han <seb@redhat.com>

We just want to set new defaults, not force the
variables. This way we can mix ceph and other
backends with like:
CINDER_ENABLED_BACKENDS=lvm:lvm-driver,ceph:ceph-driver
Change-Id: I99cc047bce2f584c2b54196a9c84267d2c686231

OpenStack CI would prefer to not have the ceph repo package mirror
enabled by default for all jobs. Instead the mirror config will be
provided in a directory that apt won't read and can by copied/linked
into the dir that apt will read.
This gives testing greater flexibility and prevents growing dependencies
on non standard repos in jobs that don't intend on growing those
dependencies.
Change-Id: Icd87013fa74601268c19a8b79abf6b7baeaca9db
Depends-On: I410f45064080e1bb93684899028e9af34bd555a1

Change-Id: Ia1adcad17c447c5ee03b9e79a9df341f877ce189

Currently the devstack ceph plugin uses installation,
configuration, startup, and shutdown commands that do not always
work correctly for Ceph releases after hammer or with systemd
init system.
Check the ceph version and the init system and use appropriate
commands.
Co-Authored-By: Ramana Raja <rraja@redhat.com>
Change-Id: I25d8816740c6008b076986f3677c299ad9843c30

Radosgw relies on shared secret token admin, passed via SERVICE_TOKEN
variable, but token admin was deprecated, and devstack doesn't
configure shared secret.
This change switch on using service credentials instead, creates
'radosgw' service user with admin privileges and configure radosgw
with them.
Change-Id: I12d4ea4021b93be73358f0aa68a32879af8fda53

Shell scripts are whitespace sensitive, in particular
a= $(echo A)
a=\
 $(echo A)
tries to run the 'A' executable somewhere in $PATH with environment variable
`a' set to an empty string.
Change-Id: I57648e597a2875061f38c8669370598348e2c6f9

In a lab setting it can be useful to allow multiple manila devstacks,
each with its own CephFS filesystem, to share a common REMOTE_CEPH
cluster.
Parameterize the name of the CephFS filesystem that devstack creates
and enable creation of multiple filesystems if appropriate variables
are set in local.conf.
Note that since separate filesystems require separate data and metadata
pools, one also needs to set values for the extant CEPHFS_DATA_POOL and
CEPHFS_METADATA_POOL parameters.
Change-Id: I67a88b75d06f04621cb0dac726f307a06ee6113b

Although the devstack ceph plugin uses variables - CEPHFS_DATA_POOL and
CEPHFS_METADATA_POOL - as arguments to the 'osd pool create' commands
that provision the pools required for manila to use the CephFS Native
backend, the overrides to the default values for these variables sets
both of them to $CEPHFS_CEPH_POOL. This setting is counter-intuitive,
and worse, forces use of a single pool for both data and metadata in the
case both default values are overridden.
Use CEPHFS_DATA_POOL and CEPHFS_METADATA_POOL as the override
variables for the data pool and metadata pool, respectively.
Change-Id: Iae2859388ce5e74b6f532cf2cf7809aaa09746cb
Closes-Bug: 1593829