openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Change-Id: Id9dbf27e93bd9a4a38a1547953be0af0bf0c19c2

The object updater emits a log line saying "Object update sweep
completed on <disk> in <time>: <lots-of-stats>" after finishing each
storage policy's async pendings. That's not right; an update sweep is
completed on a device when you've looked at all the async pendings on
the device.
This commit moves the "sweep completed" message so it's once per
device, not per policy.
Change-Id: Ibb5f02a18ac431da230afe30c556a756f60d1680

See https://review.openstack.org/#/c/571007/ and
https://review.openstack.org/#/c/570999/ for context.
Change-Id: Ib6045de3cb9097ed7e6d48646109f35612cbbc74

This patch adds a read_only middleware to swift. It gives the ability
to make an entire cluster or individual accounts read only.
When a cluster or an account is in read only mode, requests that would
result in writes to the cluser are not allowed.
DocImpact
Change-Id: I7e0743aecd60b171bbcefcc8b6e1f3fd4cef2478

Lock timeouts happen, especially on very busy containers. It's not
worth a stack trace, and it's not an error. Now we still log the
lock timeout, but at INFO level and without a stack trace.
Change-Id: Ia202c876fb726f9ae6040d875a83105dd975539b

We probably don't need an extra INFO-level message every time a user
makes a PUT request with X-Copy-From set. It's still in the proxy's
access logs.
Change-Id: I32ce8ff16cb296dd5acef07202a59bc5125111c1

Change-Id: Ia1b78aa40b1f157b513428cef59e69077b2aaee7

Change-Id: I44bb6b1702fb93d42cb6d204f631a94506257e89
Signed-off-by: Thiago da Silva <thiago@redhat.com>

Previously, these headers had to be added by operators to their
object-server.conf when enabling swift3 middleware. Since s3api
is now imported into swift we should go ahead and add these headers
by default too.
Change-Id: Ib82e175096716e42aecdab48f01f079e09da6a1d
Signed-off-by: Thiago da Silva <thiago@redhat.com>

...instead of 10,000,000. The sample configs were already using one
million, all of our testing with non-SAIO containers was done with
one million, and the resulting container DBs were around 100MB which
seems like a comfortable size. Pretty sure this was just a typo during
some code cleanup.
Change-Id: Icd31f9d8efaac2d5dc0f021cad550687859558b9

Change-Id: I3d598ad6d7791df2bc66cbaed67cd72084e0f7b3

Previously, we'd bomb out hard, and you'd get no more container updates.
Change-Id: I0cea6018269d172bc758eec6348329d1f65de028
Related-Bug: 1400939

in process of container update, when locking one container failed,
container updater stops with LockTimeout exception.
it's better for updater server to continue processing
other containers, partitions instead of stop updating.
this path is to catch timeout exception and log it,
then continue processing other containers and partitions.
Closes-bug: #1400939
Change-Id: I42adec07d980be22044a5d4ef6771318a3eed168

Also take a reasonable guess as to whether HTTPS/wsgi.url_scheme ought to be updated.
Change-Id: Iada8a1f4921f07986e6399bc1279e9a602117b1c

The mock call got pulled out in the related change.
Change-Id: Ife1e452fa34026ac934e88a05708cdc5978215e1
Related-Change-Id: Id089e29e7ecfc8cec79c520f604aa01bdae0dcf0

Fix up function tests to actually *do* that quoting, and fix
_listing_pages_iter to respect that.
Change-Id: I1554042510819ea878b4c70417721944115e17f4
Related-Bug: 1229142
Related-Change: I425440f76b8328f8e119d390bfa4c7022181e89e
Related-Bug: 1755554
Related-Change: Ibcd90cc633c68973929ee5249c6598c22b342e3e

ctypes.ssize_t exists since Python 2.7 (not just the fuzzy 2.7.X, but
CPython tag "v2.7"), so we don't need to define our own.
Change-Id: Ib3c9b162ceffabd78622ae51c5accc4b7ba1294d

Python 2.6 didn't have one, so we'd try to find logging.NullHandler
but fall back to our own. Since 2.7+ has logging.NullHandler, we can
just use it.
Change-Id: Ie2c27407efc2882e698abe6e4379a00a1d3f4301

Looks like you couldn't wrap an io.BufferedReader around a
gzip.GzipFile in 2.6, but you can starting in 2.7, so we don't need to
check for some private attribute to ensure compatibility.
Change-Id: Ib8f7378a7d8e17e0684702a9d94307d4c33cd4c8

For a while, this was conditional because we supported old Eventlet
versions that didn't have this keyword arg. Now, we require new-enough
Eventlet that it's always available, so let's get rid of the
conditional crud.
The flag was introduced in Eventlet 0.15, and we require >= 0.17.4.
Change-Id: Id089e29e7ecfc8cec79c520f604aa01bdae0dcf0

...to the proxy-server.
The point is to allow the Swift proxy server to log accurate
client IP addresses when there is a proxy or SSL-terminator between the
client and the Swift proxy server. Example servers supporting this
PROXY protocol:
 stud (v1 only)
 stunnel
 haproxy
 hitch (v2 only)
 varnish
See http://www.haproxy.org/download/1.7/doc/proxy-protocol.txt
The feature is enabled by adding this to your proxy config file:
 [app:proxy-server]
 use = egg:swift#proxy
 ...
 require_proxy_protocol = true
The protocol specification states:
 The receiver MUST be configured to only receive the protocol
 described in this specification and MUST not try to guess
 whether the protocol header is present or not.
so valid deployments are:
 1) require_proxy_protocol = false (or missing; default is false)
 and NOT behind a proxy that adds or proxies existing PROXY lines.
 2) require_proxy_protocol = true
 and IS behind a proxy that adds or proxies existing PROXY lines.
Specifically, in the default configuration, one cannot send the swift
proxy PROXY lines (no change from before this patch). When this
feature is enabled, one _must_ send PROXY lines.
Change-Id: Icb88902f0a89b8d980c860be032d5e822845d03a

If an account contains non-ASCII characters, currently SLO delete code
will fail, as get_slo_segments() method receives a unicode object, but
UTF-8 encoded account name. Attempting to concatenate the strings fails
with a UnicodeError, as it tries to use the ASCII codec to decode the
UTF-8 encoded account name.
This patch allows accounts with non-ASCII characters in their names to
delete SLOs.
Change-Id: I619d41e62c16b25bd5f58d300a3dc71aa4dc75c2

Related-Change: Ie4d2816259e6c25c346976e181fb9d350f947190
Change-Id: Icd558c1f92c24724a76931f1d281a9a20122b683

Change-Id: I25bdb2020c10f4b3aede6994e9ffc3f05a054c29

In the object server's PUT, POST, and DELETE handlers, we use the
request's X-Timestamp value for checking object expiration. In the GET
and HEAD handlers, we use it if present, but default to the current
time. That way, one can still use curl to make direct object GET or
HEAD requests as before.
If one object server's clock is ahead of the proxy server's clock for
some reason, and a client makes a POST request to update X-Delete-At,
then the skewed-time object server may refuse the new X-Delete-At
value.
In a cluster where two of the three replicas for an object live on the
same time-skewed node, this can result in confusing behavior for
clients. A client can make a POST request to update X-Delete-At,
receive a 400, and then discover later that the X-Delete-At value was
updated anyway, since one object server accepted the POST and
replication spread the new metadata around.
DELETE is somewhat less confusing. The client might get a spurious 404
in the above case, but the object will still be removed.
For PUT, an object server with a slow clock might refuse to overwrite
an object with an "older" one because it believes the on-disk object
is newer than the current time.
Change-Id: I10c28f97d4c6aca1d64bef3b93506cfbb50ade30