openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Change-Id: Ia9ae0fc226dfc9b40157faebac100c10a9180c62

I saw an encryption test fail with 10s node timeout when the py36
unittests ran in the gate. I believe there is an outsized potential for
a beneficial increase in test reliability when run in the gate compared
to relatively small chance of any negative side-effect in test failure
responsiveness.
Change-Id: Ia31912828d416d84c39782222e4636a97a8bfe44

It's probably weird that StreamingPile has this interfaces that swallows
exceptions, but this seems better than hanging.
Change-Id: I8fe45c0f0d291efc84f3edf5d6b7cd116b5c7835

We previously realized we needed to do that for accounts and containers
where the consequences of treating the 404 as authoritative were more
obvious: we'd cache the non-existence which prevented writes until it
fell out of cache.
The same basic logic applies for objects, though: if we see
 (Timeout, Timeout, Timeout, 404, 404, 404)
on a triple-replica policy, we don't really have any reason to think
that a 404 is appropriate. In fact, it seems reasonably likely that
there's a thundering-herd problem where there are too many concurrent
requests for data that *definitely is there*. By responding with a 503,
we apply some back-pressure to clients, who hopefully have some
exponential backoff in their retries.
The situation gets a bit more complicated with erasure-coded data, but
the same basic principle applies. We're just more likely to have
confirmation that there *is* data out there, we just can't reconstruct
it (right now).
Note that we *still want to check* those handoffs, of course. Our
fail-in-place strategy has us replicate (and, more recently,
reconstruct) to handoffs to maintain durability; it'd be silly *not* to
look.
UpgradeImpact:
--------------
Be aware that this may cause an increase in 503 Service Unavailable
responses served by proxy-servers. However, this should more accurately
reflect the state of the system.
Co-Authored-By: Thiago da Silva <thiagodasilva@gmail.com>
Change-Id: Ia832e9bab13167948f01bc50aa8a61974ce189fb
Closes-Bug: #1837819
Related-Bug: #1833612
Related-Change: I53ed04b5de20c261ddd79c98c629580472e09961
Related-Change: Ief44ed39d97f65e4270bf73051da9a2dd0ddbaec

Change-Id: I0c3a235b8905e02d2e202215f7f0b43db295b79b
Related-Change-Id: I66f2f16d7cf0b9cb0888e3b9b1a17b9112233d21

Otherwise, we can hit a "generator already executing" error in
test/functional/test_slo.py:TestSlo.test_slo_multi_ranged_get
Also, set a *slightly* newer lower-bound for greenlet (from 2011 instead
of 2010) so the sleep() doesn't cause us to lose exception state.
Change-Id: Id458fc0a43fd71aab9f6edbacc1f0b370c9f7537

... provided they're still zero bytes.
Change-Id: I66f2f16d7cf0b9cb0888e3b9b1a17b9112233d21

Change-Id: Ib121d6c52601488647fd16dfd4f49bb4492c7a2f

Swift gate does not run integrated-gate jobs which are
tempest-full and grenade-py3 job. Tempest-full job run
all services tests. But all services tests are not needed
to run on swift gate because it does not depends on all services.
Tempest has prepared the new jobs for swift gate which will run
only Swift, Cinder and Glance related tests.
New template 'integrated-gate-object-storage'(Depends-On) includes
the tempest-integrated-object-storage and grenade-py3 jobs
This commit adds the new integrated-gate template in swift gate.
Depends-On: https://review.opendev.org/#/c/670353/
Change-Id: Ia0543dca729c3a2aa67ce9f560c1ce5976c00abf

The URL is object-store, not object-storage. Fix one occurence.
Change-Id: I08ed3b19cbe0024fd67a6fb7a7a6db0d9084976b

Previously, we were storing the WSGI-style UTF-8-bytes-decoded-as-Latin-1
strings in the JSON field, and sending them back to eventlet directly.
If running in a mixed py2/py3 cluster, replication would eventually get
that back to the py2 server, and worse, the native-string version would
get back to the py3 server! Then on GET or HEAD, eventlet would barf
if any characters were outside the Latin-1 range.
Closes-Bug: #1837805
Change-Id: I31d942e72fd7bfbb1db4dbb1dd522dff69969e5d

Change-Id: I1152c47c52f6482ec877142c96845b00bf6dcc5b
Related-Change: I130ba5014b7eff458d87ab29eb42fe45607c9a12

The api documentation is now published on docs.openstack.org instead
of developer.openstack.org. Update all links that are changed to the
new location.
Note that the Swift API lives at /object-store and not /object-storage.
Note that redirects will be set up as well but let's point now to the
new location.
For details, see:
http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007828.html
Change-Id: Ie38357e4c278335c35d186708573bb6bdabaa012

Previously, we never checked whether the response we get when refetching
is even successful, much less whether it's still coming from an SLO.
Now, if the refetched data is newer, act on it. If it's older, 503.
Closes-Bug: #1837270
Change-Id: I106b94c77da220c762869aa800c31b87c3dffeeb

Change-Id: I2929ce1f8e08ae5b4059605d09189c8521852423

Now that os-testr has had a release, it passes again.
Change-Id: Ieed88646d97ab2a1c7323a147ee97dae6fc86a0f

Change-Id: I196b4e886942eccd2cb3aff3819da1558ddb19d4

Change-Id: I09c099339d02943baa0944b5a208263536d1a547

The contract for ReplConnection.replicate() is that if we can get a
response, we return it, and if we can't (because of a timeout, or a
socket error, or some other http_client error like BadStatusLine), we
return None to indicate the error.
Previously, _fetch_and_merge_shard_ranges() always assumed the
response would have a `status` attribute and raise an AttributeError
when response was None. Now it will treat that case like other
get_shard_range failures.
Change-Id: I023b8a46c06e9a2755b5aa890a7992ef9633cba9

Previously, we issued a GET to the root container for every object PUT,
POST, and DELETE. This puts load on the container server, potentially
leading to timeouts, error limiting, and erroneous 404s (!).
Now, cache the complete set of 'updating' shards, and find the shard for
this particular update in the proxy. Add a new config option,
recheck_updating_shard_ranges, to control the cache time; it defaults to
one hour. Set to 0 to fall back to previous behavior.
Note that we should be able to tolerate stale shard data just fine; we
already have to worry about async pendings that got written down with
one shard but may not get processed until that shard has itself sharded
or shrunk into another shard.
Also note that memcache has a default value limit of 1MiB, which may be
exceeded if a container has thousands of shards. In that case, set()
will act like a delete(), causing increased memcache churn but otherwise
preserving existing behavior. In the future, we may want to add support
for gzipping the cached shard ranges as they should compress well.
Change-Id: Ic7a732146ea19a47669114ad5dbee0bacbe66919
Closes-Bug: 1781291

Beginning with the Queens release, the keystone install guide
recommends running all interfaces on the same port.This patch
updates the swift install guide to reflect that change
Change-Id: Id00cfd2c921da352abdbbbb6668b921f3cb31a1a
Closes-bug: #1754104 

This fixes newly-enqueued work going forward, but doesn't offer anything
to try to parse existing bad work items or even to kick shards so they
reset their reconciler high-water marks.
Change-Id: I79d20209cea70a6447c4e94941e5e854889cbec5
Closes-Bug: 1836082

Change-Id: I846e746c2fe7591a3ab502428f587e3cbe753225