openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Specify that the relink command must be run as the same user than the
daemon processes to avoid creating files that are not manipulable by the
server/replicator/...
Change-Id: If23592855db5f5bb0ec1e7c679de15769fd86871

Change-Id: I1611e34846e586703e9d3709fa64e8df41f2d685

Users have complained for a while that Swift's ETags don't match the
expected RFC formats. We've resisted fixing this for just as long,
worrying that the fix would break innumerable clients that expect the
value to be a hex-encoded MD5 digest and *nothing else*.
But, users keep asking for it, and some consumers (including some CDNs)
break if we *don't* have quoted etags -- so, let's make it an option.
With this middleware, Swift users can set metadata per-account or even
per-container to explicitly request RFC compliant etags or not. Swift
operators also get an option to change the default behavior
cluster-wide; it defaults to the old, non-compliant format.
See also:
 - https://tools.ietf.org/html/rfc2616#section-3.11
 - https://tools.ietf.org/html/rfc7232#section-2.3
Closes-Bug: 1099087
Closes-Bug: 1424614
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: I380c6e34949d857158e11eb428b3eda9975d855d

This patch adds a new object versioning mode. This new mode provides
a new set of APIs for users to interact with older versions of an
object. It also changes the naming scheme of older versions and adds
a version-id to each object.
This new mode is not backwards compatible or interchangeable with the
other two modes (i.e., stack and history), especially due to the changes
in the namimg scheme of older versions. This new mode will also serve
as a foundation for adding S3 versioning compatibility in the s3api
middleware.
Note that this does not (yet) support using a versioned container as
a source in container-sync. Container sync should be enhanced to sync
previous versions of objects.
Change-Id: Ic7d39ba425ca324eeb4543a2ce8d03428e2225a1
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Thiago da Silva <thiagodasilva@gmail.com>

This allows static symlinks to be synced before their target. Dynamic
symlinks could already be synced even if target object has not been
synced, but static links previously required that target object existed
before it can be PUT. Now, have container_sync middleware plumb in an
override like it does for SLO.
Change-Id: I3bfc62b77b247003adcee6bd4d374168bfd6707d

Change-Id: I98ebe15353d675ca00fee387003bf6572ac385e6

This is to fix the duplicated words issue, like:
 * "both of which which must be stripped off"
 * "In addition the the values set"
 and so on
Change-Id: Id3d84281f15815b4185c76874575e91a3589981b

If we move it to constraints it's more globally accessible in our code,
but more importantly it's more obvious to ops that everything breaks if
you try to mis-configure different values per-service.
Change-Id: Ib8f7d08bc48da12be5671abe91a17ae2b49ecfee

Change-Id: I89f3a4b5a3a8c160afb298aad726acce09c65265

Change-Id: Ib54a65a920fc335c1e0a496cb91c0de3003e2cc5

We get a handful more passing tests that way, following their move to boto3.
Change-Id: I73e9c38bde00a7117cec97e98f013e86350aa5be

Change-Id: I775a03e0ba1e10982eb6f7ef52be773c8831b1ec

Change-Id: I3833843cd8d23c498a2afe6c68a3f0afe26343c0

Swift servers can now be seamlessly reloaded by sending them a SIGUSR1
(instead of a SIGHUP). The server forks off a synchronized child to
wait to close the old listen socket(s) until the new server has started
up and bound its listen socket(s). The new server is exec'ed from the
old one so its PID doesn't change. This makes Systemd happier, so a
ReloadExec= stanza can now be used.
The seamless part means that incoming connections will alwyas get
accepted either by the old server or the new one. This eliminates
client-perceived "downtime" during server reloads, while allowing the
server to fully reload, re-reading configuration, becoming a fresh
Python interpreter instance, etc. The SO_REUSEPORT socket option has
already been getting used, so nothing had to change there.
This patch also includes a non-invasive fix for a current eventlet bug;
see https://github.com/eventlet/eventlet/pull/590
That bug prevents a SIGHUP "reload" from properly servicing existing
requests before old worker processes close sockets and exit. The
existing probtests missed this, but the new ones, in this patch, caught
it.
New probe tests cover both old SIGHUP "reload" behavior as well as the
new SIGUSR1 seamless reload behavior.
Change-Id: I3e5229d2fb04be67e53533ff65b0870038accbb7

During a PUT of an object, the proxy instanciates one Putter per
object-server that will store data (either the full object or a
fragment, depending on the storage policy). Each Putter is owning a
Queue that will be used to bufferize data chunks before they are
written to the socket connected to the object-server. The chunks are
moved from the queue to the socket by a greenthread. There is one
greenthread per Putter. If the client is uploading faster than the
object-servers can manage, the Queue could grow and consume a lot of
memory. To avoid that, the queue is bounded (default: 10). Having a
bounded queue also allows to ensure that all object-servers will get
the data at the same rate because if one queue is full, the
greenthread reading from the client socket will block when trying to
write to the queue. So the global rate is the one of the slowest
object-server.
The thing is, every operating system manages socket buffers for incoming
and outgoing data. Concerning the send buffer, the behavior is such that
if the buffer is full, a call to write() will block, otherwise the call
will return immediately. It behaves a lot like the Putter's Queue,
except that the size of the buffer is dynamic so it adapts itself to the
speed of the receiver.
Thus, managing a queue in addition to the socket send buffer is a
duplicate queueing/buffering that provides no interest but is, as shown
by profiling and benchmarks, very CPU costly.
This patch removes the queuing mecanism. Instead, the greenthread
reading data from the client will directly write to the socket. If an
object-server is getting slow, the buffer will fulfill, blocking the
reader greenthread. Benchmark shows a CPU consumption reduction of more
than 30% will the observed rate for an upload is increasing by about
45%.
Change-Id: Icf8f800cb25096f93d3faa1e6ec091eb29500758

The Swift AIO documentation is out of date for CentOS and Fedora.
This patch updates the documentation to ensure that the instructions are
clear and accurate.
It also cleans up a few other sections along the way and adds some new
headings to make it easier to read. Some explanatory notes are added,
such as the need for XFS storage and test directories. XFS tmp loopback
device is moved out of the user's home directory to /srv to match
storage.
Change-Id: Ieb0341536b7149c99139a1cf620828eba25f4bc6
Signed-off-by: Chris Smart <chris.smart@humanservices.gov.au>

Change-Id: I948d65d0d3b58fcb13f8141fe095ccad4b8f1425
Closes-Bug: #1848485 

This patch adds a `pdf-docs` tox target that will build
PDF versions of our docs. As per the Train community goal:
 https://governance.openstack.org/tc/goals/selected/train/pdf-doc-generation.html
Add sphinxcontrib-svg2pdfconverter to doc/requirements.txt
to convert our SVGs.
Story: 2006122
Task: 35515
Change-Id: I26cefda80d3234df68d7152b404e0a71da74ab90

Change-Id: I94050c40585b397a9f7bab1e48650b89f70ab24d

Set copyright year and html_last_updated_fmt to SOURCE_DATE_EPOCH if
it's set. See https://reproducible-builds.org/specs/source-date-epoch/
This patch make build reproducible, see https://reproducible-builds.org/
Change-Id: I730a8265ca2c70c639ef77a613908e84eb738b70

Small changes, but helpful, mostly.
Backport: stein rocky
Change-Id: Ic4b6524d7804d2f74b2973b6acdb9e2679209cd4

Change-Id: I3fee824e4fdc21aac21a6bb97f24a82d8ea3577d

The URL is object-store, not object-storage. Fix one occurence.
Change-Id: I08ed3b19cbe0024fd67a6fb7a7a6db0d9084976b

The api documentation is now published on docs.openstack.org instead
of developer.openstack.org. Update all links that are changed to the
new location.
Note that the Swift API lives at /object-store and not /object-storage.
Note that redirects will be set up as well but let's point now to the
new location.
For details, see:
http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007828.html
Change-Id: Ie38357e4c278335c35d186708573bb6bdabaa012

Beginning with the Queens release, the keystone install guide
recommends running all interfaces on the same port.This patch
updates the swift install guide to reflect that change
Change-Id: Id00cfd2c921da352abdbbbb6668b921f3cb31a1a
Closes-bug: #1754104 

XFS no longer supports nobarrier mount option.
It has been deprecated for a long time[1] and removed in
recent kernel versions resulting in an error when trying to
mount: "kernel: XFS (loop0): unknown mount option [nobarrier]."
[1] - https://patchwork.kernel.org/patch/9486549/
Change-Id: Iaa9208fb20545ae9ac990f0e180899108d983123

Change-Id: Ie759c4092408a210d32f6457142457087343d33a

...to pick up many improvements, including the return of table borders.
Change-Id: I166211b690b08521171b489582fa419d756b1972

This precludes us from landing anything in the gate, because we
treat Sphinx warnings as errors in OpenStack.
Specifically, 3 warnings are fixed:
/home/zuul/src/opendev.org/openstack/swift/swift/common/middleware\
/s3api/etree.py:docstring of\
 swift.common.middleware.s3api.etree.Element:1:\
Inline strong start-string without end-string.
duplicate object description of\
 swift.common.middleware.versioned_writes, other instance in\
 middleware, use :noindex: for one of them
duplicate object description of\
 swift.common.ring.composite_builder, other instance in\
 overview_ring, use :noindex: for one of them
Change-Id: I5e0bd61a301d46b2674b4605882462575b635220

In the main branch, I think it is necessary to modify
the configuration file path in the document as the
main branch path, and to change the path to a stable
version when the version is released.
Change-Id: Ib1bc4c59ed4c7624782b8b1cdea20847c14c90bf

Missing proxy-server.conf configuration document for installing
control nodes in Debian
Change-Id: I174ba8841ed638732bcf933602c25aad8e62f0ca

Adds the scaffolding required for tests to use boto3 and converts the
test_bucket.py tests to the new interface. Follow on patches will
convert the other tests to use the boto3 library.
Notable changes: we no longer try to reach for the equivalent of
`boto.make_request()` and instead rely on the boto3/botocore event
system to mutate requests as necessary (or to disable pre-flight
validators).
Partial-Bug: 1557260
Change-Id: I3d77ef4a6b878c49ebfa0c8b8647d7199d87601e

S3 supports two metadata operations on object copy: COPY and REPLACE.
When using REPLACE, the Content-Type should be set to the one supplied
by the caller. When using COPY, the existing object's Content-Type value
is used.
Change-Id: Ic7c6278dedef308c9219eb45751abfa5655f144f
Closes-Bug: #1828907 

Added a test for S3 v1 listings that use URL encoding and have non-ASCII
characters. In the process discovered that the XML schema for
ListBucketResult had a small problem: Delimiter and EncodingType needed
to be reordered.
Change-Id: Ib3124ea079a73a577b86de97657603a64b16f965

To prepare for object-expirer's general task queue feature [1],
this patch enables to configure object-expirer in object-server.conf.
Object-expirer.conf can be used in the same manner as before, but deprecated.
If both of object-server.conf with "object-expirer" section and
object-expirer.conf are in a node, only object-server.conf is used.
Object-expirer.conf is used only if all object-server.conf doesn't have
"object-expirer" section.
There are two differences between "object-expirer.conf" style and
"object-server.conf" style.
The first difference is `dequeue_from_legacy` default value.
`dequeue_from_legacy` defines task queue mode. In "object-expirer.conf"
style, the default mode is legacy queue. In "object-server.conf" style,
the default mode is general queue. But general mode means no-op mode
for now, because general task queue is not implemented yet.
The second difference is internal client config. In "object-expirer.conf"
style, config file of internal client is the object-expirer.conf itself.
In "object-server.conf" style, config file of internal client is
another file.
[1]: https://review.openstack.org/#/c/517389/
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Change-Id: Ib21568f9b9d8547da87a99d65ae73a550e9c3230

Add the log_msg_template option in proxy-server.conf and log_format in
a/c/o-server.conf. It is a string parsable by Python's format()
function. Some fields containing user data might be anonymized by using
log_anonymization_method and log_anonymization_salt.
Change-Id: I29e30ef45fe3f8a026e7897127ffae08a6a80cd9

Thorough replacement of git.openstack.org URLs with their opendev.org
counterparts.
Change-Id: I5e77307de6a484fd69b2a5863423ceb357e8601f

It was a recommended practice for years to get away from straight
names in /dev, like /dev/sda1, when mounting filesystems. The
man page for mount(8) says:
 The device name of disk partitions are unstable; hardware
 reconfiguration, adding or removing a device can cause change
 in names. This is reason why it's strongly recommended to use
 filesystem or partition identificators like UUID or LABEL.
Nonetheless, novice operators sometimes follow our deployment
guide to the letter and then get into trouble when device names
shift from under their deployments. This patch fixes the problem
without bloating up the guide with general explanations.
Change-Id: I5faae158b62e0395d6e774cd67bd868c785c2186