openstack/swift - swift - OpenDev: Free Software Needs Free Tools

The object server can be configured to leave a certain amount of disk
space free; default is 1%. This is useful in avoiding 100%-full
filesystems, as those can get Swift in a state where the filesystem is
too full to write tombstones, so you can't delete objects to free up
space.
When a cluster has accounts/containers and objects on the same disks,
then you can wind up with a 100%-full disk since account and container
servers don't respect fallocate_reserve. This commit makes account and
container servers respect fallocate_reserve so that disks shared
between account/container and object rings won't get 100% full.
When a disk's free space falls below the configured reserve, account
and container PUT, POST, and REPLICATE requests will fail with a 507
status code. These are the operations that can significantly increase
the disk space used by a given database.
I called the parameter "fallocate_reserve" for consistency with the
object server. No actual fallocate() call happens under Swift's
control in the account or container servers (sqlite3 might make such a
call, but it's out of our hands).
Change-Id: I083442eef14bf83c0ea717b1decb3e6b56dbf1d0

Change-Id: Ie34ce2a33f2a642b97986fa28cf9db9e6da964d5
Related-Change: I67478923619b00ec1a37d56b6fec6a218453dafc
Related-Change: Ibaa630b5b4251cc4f821c01d3c09a8b8a6be342c

Related-Change-Id: I67478923619b00ec1a37d56b6fec6a218453dafc
Change-Id: Ibaa630b5b4251cc4f821c01d3c09a8b8a6be342c

Container servers will store an etag like
 <MD5 of manifest on disk>; slo_etag=<MD5 on concatenated ETags>
which the SLO middleware will break out into separate
 "hash": "<MD5 of manifest on disk",
 "slo_etag": "\"<MD5 of concatenated ETags\"",
keys for JSON listings. Text and XML listings are unaffected.
If a middleware left of SLO already specified a container update
override, the slo_etag parameter will be appended. If the base header
value was blank, the MD5 of the manifest will be inserted.
SLOs that were created on previous versions of Swift will continue to
just have the MD5 of the manifest in container listings.
Closes-Bug: 1618573
Change-Id: I67478923619b00ec1a37d56b6fec6a218453dafc

If swift-ring-builder is erroneously given a composite builder
file, which it will fail to load, it will now print a hint
that the file is a composite builder file.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: If4517f3b61977a7f6ca3e08ed5deb182aa87a366

This patch adds an additional optional parameter to tempurl
which restricts the ip's from which a temp url can be used from.
Change-Id: I23fe998a980960d4a32df042b3f6a21f096c36af

Add a new middleware that can be used to fetch an encryption root
secret from a KMIP service. The middleware uses a PyKMIP client
to interact with a KMIP endpoint. The middleware is configured with
a unique identifier for the key to be fetched and options required
for the PyKMIP client.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: Ib0943fb934b347060fc66c091673a33bcfac0a6d

In test_obj.py there are PUT tests that are specific to the Putter
type and others that apply to any Putter type. This patch refactors
them into separate classes to provide greater clarity and to allow
common tests to be applied to each Putter type.
Taking this infrastracture out and ahead of PUT+POST itself allows
it to be reviewed much easier, especially when you look at the diff.
Related-Change-Id: I64b0d8fdb2ffce786f56665a74ed7eb2603abfda
Change-Id: Ibb09b5a28098fb51e25ab5a7134b518cc68eaf89

Yes, this means you can now start a proxy-server process on py3! And it
will give you something useful on /info!
Apparently you can even get auth tokens from tempauth, although the
tempauth unit tests don't pass yet.
Change-Id: I86ead2989b5934a7584cdd75719ce239826e01ec

Instead, require that callers provide an encoding.
Related-Change: I31408f525ba9836f634a35581d4aee6fa2c9428f
Change-Id: I3e5ed9e4401eea76c375bb43ad4afc58b1d8006a

There's already a nice way to reap a process if it's exited while not
waiting, and it doesn't require timeouts.
Change-Id: Ie327fecb6a3055ff146a94e1171ec0ec68d7179f
Related-Change: If6dc7b003e18ab4e8a5ed687c965025ebd417dfa

If we're going to drop the quotes during initialization, we should
probably drop them when making comparisons, too.
Change-Id: I90c6e8a7c6faf8fafb0cd64fabbc9629d6c4c48a

Change-Id: Iee1bab5775b243c318aa22ee4a548d793e6684ca

The domain_remap middleware didn't need to change, only its tests.
Change-Id: I5edf05b609ce263708f851219c38c1bd1b645a6f

Typed "b" in a half-dozen places and fixed a typo in a test name.
Change-Id: Idcb8b2eeef5b545eb0944cc22b7c7961e8c2f8f0

Change-Id: I83a35f066d69edf39c74fd22c7fb248b5320d21f

The ignored_secret variable wasn't being used.
Change-Id: Ia659f3672941f9bb86054ce6cafbbe6fe7ccafbc
Related-Change: I1cd9a97c4e8d87d7c065866e7ad3a9e748ff19ab

Otherwise, a sharded container AUTH_test/sharded will have its stats
included in the totals for both AUTH_test *and* .shards_AUTH_test
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I7fa74e13347601c5f44fd7e6cf65656cc3ebc2c5

Today's diff is brought to you by the letter "b".
Change-Id: I074c1ff3f1232a899c101f2856dff4a881a6fe77

Only the tests needed to change, and it was nearly all just prepending
"b" to a bunch of string literals.
Change-Id: Id5317e471c04a05627438c281bb0d6a97cf6a8a4

Change-Id: Ibdb9e2bbec1c962d930a3f69fc95a8c562ac13b7

...which helps us differentiate between a drive that's not mounted vs.
not a dir better in log messages. We were already doing that a bit in
diskfile.py, and it seems like a useful distinction; let's do it more.
While we're at it, remove some log translations.
Related-Change: I941ffbc568ebfa5964d49964dc20c382a5e2ec2a
Related-Change: I3362a6ebff423016bb367b4b6b322bb41ae08764
Change-Id: Ife0d34f9482adb4524d1ab1fe6c335c6b287c2fd
Partial-Bug: 1674543

Change-Id: I31408f525ba9836f634a35581d4aee6fa2c9428f

Alternate approach to https://review.openstack.org/#/c/559902
Change-Id: I1e244c231753b8f4b6f1cf95cb0ae4c3c959ae0f

Resolve outstanding TODO's. One TODO is removed because there isn't an
easy way to arrange for an async pending to be targeted at a shard
container.
Change-Id: I0b003904f73461ddb995b2e6a01e92f14283278d

eventlet<=0.17.4 LoggerFileWrapper does not have an error() method
so don't try to call it. Instead, use info() which even in newer
eventlet versions does exactly the same as error().
This bug only manifests in unit tests once the Related-Bug has been fixed
(see Related-Change).
Closes-Bug: #1777836
Related-Bug: #1777701
Related-Change: I68fd9d0263a6720aaf0d90b8ea6da1158105ac01
Change-Id: I46190567549826c811ffa51e9a71a38a20d9ce97

Older versions of eventlet expect getsocketname() to return a tuple,
and newer versions are happy to receive a tuple, so fix the mock to
return a tuple.
Change-Id: I68fd9d0263a6720aaf0d90b8ea6da1158105ac01
Closes-Bug: #1777701