openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Change-Id: If347476e3b9185921ff174d3f8170a1c4d0622e8

If we're going to encapsulate the stats tracking it seems reasonable if
we ever add any more metrics we can reduce the number of places we need
to update log messages.
Change-Id: I187cf6cfec1e0a9138b709fa298e1991aa809ec4

This boils down to 404, 412, or 416; or 409 when we provided an
X-Timestamp.
This means, among other things, that the expirer won't issue 3 DELETEs
every cycle for every stale work item.
Related-Change: Icd63c80c73f864d2561e745c3154fbfda02bd0cc
Change-Id: Ie5f2d3824e040bbc76d511a54d1316c4c2503732

Change-Id: I9a04cb2fbe61e1fbd8185ab2fac9abbcea4d55cc

Why weren't we doing that before?? The etag should be the same as for
GET/HEAD, and by sending it, we can assure resuming clients that they're
downlading the same object even if they didn't include an If-Match
header.
Change-Id: I4ccbd1ae3a909ecb4606ef18211d1b868f5cad86
Related-Change: Ic11662eb5c7176fbf422a6fc87a569928d6f85a1

... in order to make the test pass with more seeds and fail less
frequently in the gate.
Change-Id: I059e80af87fd33a3b6c0731fbad62e035215eca5

The object updater has five different stats, but its logging only told
you two of them (successes and failures), and it only told you after
finishing all the async_pendings for a device. If you have a cluster
that's been sick and has millions upon millions of async_pendings
laying around, then your object-updaters are frustratingly
silent. I've seen one cluster with around 8 million async_pendings per
disk where the object-updaters only emitted stats every 12 hours.
Yes, if you have StatsD logging set up properly, you can go look at
your graphs and get real-time feedback on what it's doing. If you
don't have that, all you get is a frustrating silence.
Now, the object updater tells you all of its stats (successes,
failures, quarantines due to bad pickles, unlinks, and errors), and it
tells you incremental progress every five minutes. The logging at the
end of a pass remains and has been expanded to also include all stats.
Also included is a small change to what counts as an error: unmounted
drives no longer do. The goal is that only abnormal things count as
errors, like permission problems, malformed filenames, and so
on. These are things that should never happen, but if they do, may
require operator intervention. Drives fail, so logging an error upon
encountering an unmounted drive is not useful.
Change-Id: Idbddd507f0b633d14dffb7a9834fce93a10359ab

Since commit 1e79f828, we don't need to test with post_as_copy=True
any more since we haven't got post_as_copy at all.
Change-Id: I9c96ce0b812d877bbe11bdb50eb160d6ffa5933d

The related change [1] causes expiring object records to no longer be
created if the X-Delete-At-Container header is not sent to the object
server, but old proxies prior to [2] (i.e. releases prior to 1.9.0)
did not send this header.
The goal of [1] can be alternatively achieved by making expiring
object record creation be conditional on the X-Delete-At-Host header.
[1] Related-Change: I20fc2f42f590fda995814a2fa7ba86019f9fddc1
[2] Related-Change: Id0873a3f2198ce285fe0b0c777738eff38bc2438
Change-Id: Ia0081693f01631d3f2a59612308683e939ced76a

After deleting an object, the object expirer deletes the corresponding
row from the expirer queue by making DELETE requests directly to the
container servers. The same thing happens after attempting to delete
an object, but failing because the object has already been deleted. If
the DELETE requests fail, then the expirer will encounter that row
again on its next pass and retry the DELETE at that time. Therefore,
it is not necessary for the object server to write an async_pending
for that queue row's deletion.
Currently, however, two of the object servers do write such
async_pendings. Given Rc container replicas, that's 2 * Rc updates
from async_pendings and another Rc from the object expirer
directly. Given a typical Rc of 3, that's 9 container updates per
expiring object.
This commit makes the object server write no async_pendings for DELETE
requests coming from the object expirer. This reduces the number of
container server requests to Rc (typically 3), all issued directly
from the object expirer.
Closes-Bug: 1076202
Change-Id: Icd63c80c73f864d2561e745c3154fbfda02bd0cc

Can we go deeper!?
Change-Id: Ibd3b06542aa1bfcbcb71cc98e6bb21a6a67c12f4

test_check_delete_headers_removes_delete_after was
failing intermittently due to rounding of float time
values.
Change-Id: Ia126ad6988f387bbd2d1f5ddff0a56d457a1fc9b
Closes-Bug: #1743804 

If we don't drain the body, the proxy logging in the internal client
pipeline will log 499 client disconnect instead of actual error response
code.
For error responses, we try to do the most helpful thing using swob's
closing and caching response body attribute. For non-error responses
which are returned to the client, we endeavour to keep the app_iter
intact and unconsumed, trusting expecting the caller to do the right
thing is the only reasonable interface. We must cleanly close any WSGI
app_iter which we do not return to the client regardless of status code
and allow the logging of the 499 if needed.
Closes-Bug: #1675650
Change-Id: I455b5c38074ad0e72aa5e0b05771e193208905eb

Send just as many requests with X-Delete-At-* as we do X-Container-* to
the object server. Furthermore, stop the object server on making an
update to the expirer queue when it wasn't told to do so and remove the
log warning which would have been produced.
Reason:
It can be the case that the number of object replicas (OR) is larger
than the number of container replicas (CR) for a given storage policy
(most likely in case of EC). Before this commit, only CR object servers
received the x-delete-at-* headers, which means that OR - CR object
servers did not receive the headers. The servers missing the header
would produce a log warning and create the x-delete-at-container header
and async update on their own, which could lead to a bug, if the
expiring_objects_container_divisor option was misconfigured.
Change-Id: I20fc2f42f590fda995814a2fa7ba86019f9fddc1
Closes-Bug: #1733588 

If you want more information, you need to go check out the *other* node.
Maybe this should be further refined to only log at debug for specific
statuses like 404 and 412?
Partial-Bug: 1688558
Related-Bug: 1455221
Change-Id: Ieefd8841154faba40dcf2a03abc5f056bdccd54f

Change-Id: Ia21f8743bfd745f2579db8658624f888461c2cc2

The pointer to using COPY to the same object as a mechanism to set only
a subset of the metadata, it does not mention that doing so results in
a full copy of the object in question on the backend.
Add a note so it's clear that there is a tradeoff involved.
Change-Id: I0c20a4909a6c3ff672f753d26cb9fb2f5f33d1f4

Use https instead of http for some links in readme.rst
Change-Id: Idd382f58108e96129c69c6dc149c694fd7833fb3

...and update the object-expirer man page.
Change-Id: Idca1b8e3b7d5b40481af0d60477510e2557b88c0

... to make testing more targeted and obvious
Related-Change-Id: I89439286b211f2c5ef19deffa77c202f48f07cf8
Change-Id: I93b99128a4fb35395e8e9caf11649e216f824fdf

There are three similar code segments in rebalance process as follows:
 tiers = ['cluster', 'regions', 'zones', 'servers', 'devices']
 for i, tier_name in enumerate(tiers):
 replicas_at_tier = sum(weighted_replicas_by_tier[t] for t in
 weighted_replicas_by_tier if len(t) == i)
 if abs(self.replicas - replicas_at_tier) > 1e-10:
 raise exceptions.RingValidationError(
 '%s != %s at tier %s' % (
 replicas_at_tier, self.replicas, tier_name))
I think we can encapsulate this code segment to a private function and
replace those code segments with a function call
Change-Id: I89439286b211f2c5ef19deffa77c202f48f07cf8

Swift allows autocreate account. It should be treat as server error
instead of 404 when it fails to create account
Change-Id: I726271bc06e3c1b07a4af504c3fd7ddb789bd512
Closes-bug: 1718810

Check that when X-Backend-Clean-Expiring-Object-Queue is true
the object server does indeed call async_update.
Change-Id: I0a87979147591f15349b868a12ac6dd15ac4e37f
Related-Change: I4d64f4d1d107c437fd3c23e19160157fdafbcd42