openstack/swift - swift - OpenDev: Free Software Needs Free Tools

In the main branch, I think it is necessary to modify
the configuration file path in the document as the
main branch path, and to change the path to a stable
version when the version is released.
Change-Id: Ib1bc4c59ed4c7624782b8b1cdea20847c14c90bf

Missing proxy-server.conf configuration document for installing
control nodes in Debian
Change-Id: I174ba8841ed638732bcf933602c25aad8e62f0ca

Adds the scaffolding required for tests to use boto3 and converts the
test_bucket.py tests to the new interface. Follow on patches will
convert the other tests to use the boto3 library.
Notable changes: we no longer try to reach for the equivalent of
`boto.make_request()` and instead rely on the boto3/botocore event
system to mutate requests as necessary (or to disable pre-flight
validators).
Partial-Bug: 1557260
Change-Id: I3d77ef4a6b878c49ebfa0c8b8647d7199d87601e

S3 supports two metadata operations on object copy: COPY and REPLACE.
When using REPLACE, the Content-Type should be set to the one supplied
by the caller. When using COPY, the existing object's Content-Type value
is used.
Change-Id: Ic7c6278dedef308c9219eb45751abfa5655f144f
Closes-Bug: #1828907 

Added a test for S3 v1 listings that use URL encoding and have non-ASCII
characters. In the process discovered that the XML schema for
ListBucketResult had a small problem: Delimiter and EncodingType needed
to be reordered.
Change-Id: Ib3124ea079a73a577b86de97657603a64b16f965

To prepare for object-expirer's general task queue feature [1],
this patch enables to configure object-expirer in object-server.conf.
Object-expirer.conf can be used in the same manner as before, but deprecated.
If both of object-server.conf with "object-expirer" section and
object-expirer.conf are in a node, only object-server.conf is used.
Object-expirer.conf is used only if all object-server.conf doesn't have
"object-expirer" section.
There are two differences between "object-expirer.conf" style and
"object-server.conf" style.
The first difference is `dequeue_from_legacy` default value.
`dequeue_from_legacy` defines task queue mode. In "object-expirer.conf"
style, the default mode is legacy queue. In "object-server.conf" style,
the default mode is general queue. But general mode means no-op mode
for now, because general task queue is not implemented yet.
The second difference is internal client config. In "object-expirer.conf"
style, config file of internal client is the object-expirer.conf itself.
In "object-server.conf" style, config file of internal client is
another file.
[1]: https://review.openstack.org/#/c/517389/
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Change-Id: Ib21568f9b9d8547da87a99d65ae73a550e9c3230

Add the log_msg_template option in proxy-server.conf and log_format in
a/c/o-server.conf. It is a string parsable by Python's format()
function. Some fields containing user data might be anonymized by using
log_anonymization_method and log_anonymization_salt.
Change-Id: I29e30ef45fe3f8a026e7897127ffae08a6a80cd9

Thorough replacement of git.openstack.org URLs with their opendev.org
counterparts.
Change-Id: I5e77307de6a484fd69b2a5863423ceb357e8601f

It was a recommended practice for years to get away from straight
names in /dev, like /dev/sda1, when mounting filesystems. The
man page for mount(8) says:
 The device name of disk partitions are unstable; hardware
 reconfiguration, adding or removing a device can cause change
 in names. This is reason why it's strongly recommended to use
 filesystem or partition identificators like UUID or LABEL.
Nonetheless, novice operators sometimes follow our deployment
guide to the letter and then get into trouble when device names
shift from under their deployments. This patch fixes the problem
without bloating up the guide with general explanations.
Change-Id: I5faae158b62e0395d6e774cd67bd868c785c2186

Replace 'o' with 'to'.
Change-Id: I0a9b1547016b2662002c050e8388591d7d91ef97

Drive-by: use six.moves in s3api; fix "unexpected indent" warning when building
docs on py3
Change-Id: I2a354e2624c763a68fcea7a6404e9c2fde30d631

At the moment, the `get_data_dir` ref in [0] is not
pointing to anything. This patch amends that and
links it to the correct policy string.
Without updating this change, local tox builds
for swift documentation fail with the following
error: Warning, treated as error: /home/asettle/openstack/swift/doc
/source/overview_policies.rst:555:more than one target found for
cross-reference u'get_data_dir': swift.obj.reconstructor.get_data_dir,
swift.obj.diskfile.get_data_dir, swift.obj.replicator.get_data_dir
[0] https://docs.openstack.org/swift/rocky/overview_policies.html#object-server
Change-Id: I7c699e4fc46706a4971fce5a85ed335f471d3a2b

Change-Id: I7c716dea5e0a5b8849b84b1bb25d5294591dcd51

To increase performance of the s3 API retrieve and cache s3 secret
from keystone to allow for local validation.
Disabled by default, to use set 'secret_cache_duration' to a
number greater than 0.
You will also need to configure keystone auth credentials in
the s3token configuration group too. These credentials will
need to be able to view all projects credentials in keystone.
Change-Id: Id0c01da6aa6ca804c8f49a307b5171b87ec92228

This patch updates the overview_encryption page to add a
`Changing the encryption root secret of external KMS's` section
to point out the slight difference in naming. I.E:
 key_id_<secret_id> vs. encryption_root_secret_<secret_id>
This patch refers to both multikey support in the KMIP and KMS
key masters, so really should land after both of them.
Related-Change-Id: Ie52508e47d15ec5c4e96902d3c9f5f282d275683
Related-Change-Id: I4f485dcb31e5bea511c4e539c54681091fc5bb1c
Change-Id: Ie4cd8ae038501c8abc43d09cf0b207ca375a4366

Change-Id: Ib00193b62c1f35b06a93f261bd6cb0d2f9167b86

Change-Id: I196fc4212b5405c410c9bfb850cd0d4737094c47

Most daemons have a "go as fast as you can then sleep for 30 seconds"
strategy towards resource utilization; the object-updater and
object-auditor however have some "X_per_second" options that allow
operators much better control over how they spend their I/O budget.
This change extends that pattern into the account-replicator,
container-replicator, and container-sharder which have been known to peg
CPUs when they're not IO limited.
Partial-Bug: #1784753
Change-Id: Ib7f2497794fa2f384a1a6ab500b657c624426384

xrange is not defined in python3.
Rename xrange() to range().
Change-Id: Ifb1c9cfd863ce6dfe3cced3eca7ea8e539d8a5e9

When a bucket already exists, PUT returned a BucketAlreadyExists error.
AWS S3 returns BucketAlreadyOwnedByYou error instead, so this changes
the error returned by swift3.
When sending a PUT request to a bucket, if the bucket already exists and
is not owned by the user, return 409 conflict error,
BucketAlreadyExists.
Change-Id: I32a0a9add57ca0e4d667b5eb538dc6ea53359944
Closes-Bug: #1498231 

Change-Id: Idcb1e9cacba5b959387a2bfd7a4ef5e9d502996a

In follow-up to the related change, mention the new
cors_expose_headers option (and other proxy-server.conf
options) in the CORS doc.
Add a test for the cors options being loaded into the
proxy server.
Improve CORS comments in docs.
Change-Id: I647d8f9e9cbd98de05443638628414b1e87d1a76
Related-Change: I5ca90a052f27c98a514a96ee2299bfa1b6d46334

also a drive-by fix to the bindep job--it doesn't need to install
Change-Id: Ic9b68bc60bfbf21b45a1b7f9d7ff9998e01ddd26

Change-Id: Ic7025ba79eadb39f75dd03135fae3326138b3ded

User can cofigure KEEPIDLE time for sockets in TCP connection.
The default value is the old value which is 600.
Change-Id: Ib7fb166deb8a87ae4e97ba0671048b1ec079a2ef
Closes-Bug:1759606

Previously we'd use two users, one admin and one unprivileged.
Ceph's s3-tests, however, assume that both users should have access to
create buckets. Further, there are different errors that may be returned
depending on whether you are the *bucket* owner or not when using
s3_acl. So now we've got:
 test:tester1 (admin)
 test:tester2 (also admin)
 test:tester3 (unprivileged)
Change-Id: I0b67c53de3bcadc2c656d86131fca5f2c3114f14

This patch added new non-voting gate job to check the s3api compatibility
via swiftstack/s3compat tool that shows the ratio of compatible S3 APIs
in the gate result for each patch. This is very useful to check the possibility
if the new incoming patch breaks S3 API compatibility unexpectedly.
Originally swift3 has this kind of the gate job but we missed the staff
while migrating from swift3 into swift upstream repo so this is the porting of that.
Note that currently the job is against to only tempauth because we don't have
custom gate jobs using keystone environment other than dsvm.
Change-Id: I6f30f74678ad35479da237361bee48c46c0ecc49

Change-Id: Ib82f2494f2cc65a3f15925efe0dfeb52eb7aa22b

Change-Id: I1337712cc790baca151db2575f650da243bf09ef

With this commit, each storage policy can define the diskfile to use to
access objects. Selection of the diskfile is done in swift.conf.
Example:
 [storage-policy:0]
 name = gold
 policy_type = replication
 default = yes
 diskfile = egg:swift#replication.fs
The diskfile configuration item accepts the same format than middlewares
declaration: [[scheme:]egg_name#]entry_point
The egg_name is optional and default to "swift". The scheme is optional
and default to the only valid value "egg". The upstream entry points are
"replication.fs" and "erasure_coding.fs".
Co-Authored-By: Alexandre Lécuyer <alexandre.lecuyer@corp.ovh.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I070c21bc1eaf1c71ac0652cec9e813cadcc14851

Seen locally:
 Warning, treated as error:
 .../swift/doc/source/overview_policies.rst:555:more than one target
 found for cross-reference u'get_data_dir':
 swift.obj.reconstructor.get_data_dir, swift.obj.replicator.get_data_dir,
 swift.obj.diskfile.get_data_dir
Not sure why it hasn't been seen in the gate...
The whole sentence is suspect, though; the Diskfile class doesn't define
a get_data_dir method, though it uses the module-level get_data_dir...
Change-Id: I6855c82315e1c71596ecce25b66b54133c239377

For some use cases operators would like to periodically introduce a
new encryption root secret that would be used when new object data is
written. However, existing encrypted data does not need to be
re-encrypted with keys derived from the new root secret. Older root
secret(s) would still be used as necessary to decrypt older object
data.
This patch modifies the KeyMaster class to support multiple root
secrets indexed via unique secret_id's, and to store the id of the
root secret used for an encryption operation in the crypto meta. The
decrypter is modified to fetch appropriate keys based on the secret id
in retrieved crypto meta.
The changes are backwards compatible with previous crypto middleware
configurations and existing encrypted object data.
Change-Id: I40307acf39b6c1cc9921f711a8da55d03924d232

Added healthcheck middleware to account, container, object servers
Added the s3api, keymaster, encryption config to the proxy config
file to make it easy to enable it.
Change-Id: I96f120c5bc416e9aba388cbfa6c30b648d6ade2f

These files are imported (and very lightly edited) from the old
ocata user-guide. It has a few other swift-related docs that seemed
more duplacative of what we already have, but these seem to fill
existing gaps in our docs.
Change-Id: Ib00bf6992327f15f271120dc5dbc86a4a235baec

The object server can be configured to leave a certain amount of disk
space free; default is 1%. This is useful in avoiding 100%-full
filesystems, as those can get Swift in a state where the filesystem is
too full to write tombstones, so you can't delete objects to free up
space.
When a cluster has accounts/containers and objects on the same disks,
then you can wind up with a 100%-full disk since account and container
servers don't respect fallocate_reserve. This commit makes account and
container servers respect fallocate_reserve so that disks shared
between account/container and object rings won't get 100% full.
When a disk's free space falls below the configured reserve, account
and container PUT, POST, and REPLICATE requests will fail with a 507
status code. These are the operations that can significantly increase
the disk space used by a given database.
I called the parameter "fallocate_reserve" for consistency with the
object server. No actual fallocate() call happens under Swift's
control in the account or container servers (sqlite3 might make such a
call, but it's out of our hands).
Change-Id: I083442eef14bf83c0ea717b1decb3e6b56dbf1d0

Change-Id: Ifd2f468ad745d19ef474ae7503a8bd79d429fc1b