openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Use Timestamp.internal and Timestamp.normal methods to better clarify
which timestamp format is used with which headers.
Use setUp provided Timestamp iters more.
Change-Id: I75b4d2fb946227fb461e8a7512f1d0f9bd3e6d9f
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

There's many places where a Last-Modified header value is constructed
from a time value. This patch replaces the duplicate code with a
helper function that returns the correctly formatted header for a
given number of seconds.
Change-Id: Ia7c619fe563ab89ff5b1beb03219e763a8798cc9
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

An object server unit test was seen to fail returning a 409 when a 204
was expected. I cannot see the bug, so this patch adds some failure
context to add illumination if/when it fails again.
Change-Id: I45327cb5a56a87b8f6b9e971232fdce3b3f4bcb8
Related-Bug: #2131684
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

If a checksum header is sent with an S3 API DeleteObjects POST request
then it will be verified (unlike a CompleteMultipartUpload POST) [0].
If a checksum header is not sent with an S3 API DeleteObjects POST
request then a Content-MD5 header is required. Even when not
*required* [1], a Content-MD5 value will be verified if sent.
Previously we could return 200 when we should have sent 400 (eg, when
the client sent a sha256 but no md5 or additional checksum), or return
400 when we should have sent 200 (eg, when an additional checksum was
sent with no sha256 and the auth protocol didn't require a sha256).
[0] Related-Change: Id39fd71bc59875a5b88d1d012542136acf880019
[1] Related-Change: Ifbcde9820bee72d80cab0fe3e67ea0f5817df949
Related-Bug: #2131671
Change-Id: If25a8f0a3079558544ab15c874eda666a9f69933
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
Co-authored-by: Tim Burke <tim.burke@gmail.com>

Clarify that the caught exception message is not used to construct the
resulting log message.
Change-Id: I5cdaa27d25f43865eab2f8b12cc68a5d086ef0c7
Related-Change: Ib54172131f1e3abd9b0419aa0370930c6fc82ba9
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Recent versions of Keystone require auth tokens when accessing the
/v3/s3tokens endpoint to prevent exposure of a lot of information that
a user who just has a presigned URL should not be able to see.
UpgradeImpact
=============
The s3token middleware now requires Keystone auth credentials to be
configured. If secret_cache_duration is enabled, these credentials
should already be configured. Without these credentials, Keystone users
will no longer be able to make S3 API requests.
Closes-Bug: #2119646
Change-Id: Ie80bc33d0d9de17ca6eaad3b43628724538001f6
Signed-off-by: Tim Burke <tim.burke@gmail.com>

Currently we can get one proxy-logging transfer stat emission over the
duration of the upload/download. We want another stat coming out of
proxy-logging: something that gets emitted periodically as bytes are
actually sent/received so we can get reasonably accurate point-in-time
breakdowns of bandwidth usage.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Shreeya Deshpande <shreeyad@nvidia.com>
Change-Id: Ideecd0aa58ddf091c9f25f15022a9066088f532b
Signed-off-by: Yan Xiao <yanxiao@nvidia.com>

AFAIK this is un-needed once we ensure test_db_replicator un-patches the
object with a cleanup, but in isolation a test patching a global and
relying on someting else to clean it up looks messy/lazy - or at best
just not a pattern/style that can easily be copied elsewhere.
Related-Change: I66d4df1e2dba058b7c719a4a932234b3fc10b554
Change-Id: Ib54172131f1e3abd9b0419aa0370930c6fc82ba9
Signed-off-by: Clay Gerrard <clay.gerrard@gmail.com>

AFAIK this isn't causing any problems, but looks weird once we fix the
db_replicator.ReplConnection patching in Related-Change.
Related-Change: I66d4df1e2dba058b7c719a4a932234b3fc10b554
Change-Id: I45b10c71bab6933dd32c042e386f59e37279787b
Signed-off-by: Clay Gerrard <clay.gerrard@gmail.com>

replace a questionable reload_module with idiomatic addCleanup
Change-Id: I66d4df1e2dba058b7c719a4a932234b3fc10b554
Signed-off-by: Clay Gerrard <clay.gerrard@gmail.com>

get_s3_access_key_id returns the S3 access_key_id user for the request
is_s3_req checks whether a request looks like it ought to be an S3 request
parse_path returns a wsgi string
extract_bucket_and_key extracts bucket and object from the request's PATH_INFO
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Shreeya Deshpande <shreeyad@nvidia.com>
Change-Id: Iaf86a07238cca6700dee736f55d4c0672cccf1b1
Signed-off-by: Shreeya Deshpande <shreeyad@nvidia.com>

Change-Id: I1e53d171faff02e2dbcbcc779ad3a47506b26853
Signed-off-by: Clay Gerrard <clay.gerrard@gmail.com>

* add a self.temp_dir in setUp and remove in teraDown
* be consistent in order of (expected, actual) args
* assert the complete exception error line
Related-Change: I289d3e9b6fe14159925786732ad748acd0459812
Change-Id: I185c8cd55db6df593bb3304c54c5160c1f662b86
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

The relinker has already been robust to hardlink collisions on
tombstones for some time; this change allows ops to optionally
(non-default) enable a similar handling of other files when relinking
the old=>new partdir.
If your cluster is having a bunch of these kinds of collisions and after
spot checking you determine the data is in fact duplicate copies the
same data - you'd much rather have the option for the relinker to
programatically handle them non-destructively than forcing ops to rm a
bunch of files manually just get out of a PPI.
Once the PPI is over and you reconstrcutors are running again, after
some validation you can probably clean out your quarantine dirs.
Drive-by: log unknown relink errors at error level to match expected
non-zero return code
Closes-Bug: #2127779
Change-Id: Iaae0d9fb7a1949d1aad9aa77b0daeb249fb471b5
Signed-off-by: Clay Gerrard <clay.gerrard@gmail.com>

Extract helpers (from container.sharder) that formats log
context from either a broker (preferring broker.path/db_file)
or a plain db_file string. Use it in common.db_replicator and
container.replicator so messages are uniform and robust.
Update tests to cover both cases.
No functional changes to replication behavior; this is
logging/robustness and test updates only.
Change-Id: I289d3e9b6fe14159925786732ad748acd0459812
Related-Change: I7d2fe064175f002055054a72f348b87dc396772b
Signed-off-by: ashnair <ashnair@nvidia.com>

Theoretically, the various WSGI servers should be able to operate on a
system without /etc/swift/swift.conf. However, this doesn't actually
work. WSGI servers call utils.validate_configuration() before looking
for a swift_dir option, and that validation reads swift.conf from its
default location. Even if you set swift_dir=/some/where/else, the WSGI
servers require /etc/swift/swift.conf to exist.
This commit makes the WSGI servers call utils.set_swift_dir before
calling utils.validate_configuration.
Motivation: I'm working on testing some client software against actual
Swift, but my CI environment doesn't have /etc/swift at all, so the
test suite can't start the Swift daemons.
Change-Id: Ie0efee33e684b1c5bad6ee2191c187bb680de5f1
Signed-off-by: Samuel Merritt <smerritt@nvidia.com>

Move tests to base TestCase, currently they are only running against old
"broker w/o metadata" - but the tests and behavior should work on all
versions of the account schema
Drive-by: reword tests to make assertions stronger and behaviors more
obvious
Related-Change: Ic7c2aa878caf039b29abb900b4f491130be3d8a8
Change-Id: I59abd956ffa01bd41f29959ff3df89a3a20a00d4
Signed-off-by: Clay Gerrard <clay.gerrard@gmail.com>

iterate through not allowed methods and assert
Change-Id: Ia304709fc56d3e81bb1326b56a4b0d64ed698160
Signed-off-by: Tim Burke <tim.burke@gmail.com>

In addition to being in the pure unmolested ondisk format from the
source container; the manifest must also include the normally
protected X-Static-Large-Object metadata.
Change-Id: Ic6638e8258e9dec755f8d9630f0586bd3c9b4420
Related-Change: I8d503419b7996721a671ed6b2795224775a7d8c6
Signed-off-by: Tim Burke <tim.burke@gmail.com>

Related-Change-Id: I38c11b7aae8c4112bb3d671fa96012ab0c44d5a2
Change-Id: Ibe5d206d2b96e174f849715fb13562ae0d2f5de2
Signed-off-by: Clay Gerrard <clay.gerrard@gmail.com>

This patch replaces --force-commits with a --skip-commits flag in
swift-manage-shard-ranges to determine when to commit object updates.
Change-Id: I6de041f5c12dca2618d22d1271efe242b2f35258
Signed-off-by: Christian Ohanaja <cohanaja@nvidia.com>

The cost of memcache misses could be deadly. For example, when
updating shard range cache query miss, PUT requests would have to
query the backend to figure out which shard to upload the objects.
And when a lot of requests are sending to the backend at the same
time, this could easily overload the root containers and cause a
lot of 500/503 errors; and when proxy-servers receive responses of
all those 200 backend shard range queries, they could in turn try
to write the same shard range data into memcached servers at the
same time, and cause memcached to return OOM failures too.
We have seen cache misses frequently to updating shard range cache
in production, due to Memcached out-of-memory and cache evictions.
To cope with those kind of situations, a memcached based cooperative
token mechanism can be added into proxy-server to coalesce lots of
in-flight backend requests into a few: when updating shard range
cache misses, only the first few of requests will get global
cooperative tokens and then be able to fetch updating shard ranges
from backend container servers. And the following cache miss
requests will wait for cache filling to finish, instead of all
querying the backend container servers. This will prevent a flood
of backend requests to overload both container servers and memcached
servers.
Drive-by fix: when memcache is not available, object controller will
only need to retrieve a specific shard range from the container server
to send the update request to.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Yan Xiao <yanxiao@nvidia.com>
Co-Authored-By: Shreeya Deshpande <shreeyad@nvidia.com>
Signed-off-by: Jianjian Huo <jhuo@nvidia.com>
Change-Id: I38c11b7aae8c4112bb3d671fa96012ab0c44d5a2

Add a path property for AccountBroker and use lazy, resilient
_populate_instance_cache(). Use None attrs as flags, avoid broad
try/except in path, and retry if cache population fails.
Change-Id: Ic7c2aa878caf039b29abb900b4f491130be3d8a8
Signed-off-by: ashnair <ashnair@nvidia.com>

Memcached based cooperative token is a improved version of ghetto lock,
see the description of ghetto lock at here:
https://github.com/memcached/memcached/wiki/ProgrammingTricks
It's used to avoid the thundering herd situation which many caching
users face: given a cache item that is popular and difficult to
recreate, in the event of cache misses, users could end up with hundreds
(or thousands) of processes slamming the backend database at the same
time in an attempt to refill the same cache content. This thundering
herd problem not only often leads to unresponsive backend; and also
those writes into memcached cause premature cache eviction under memory
pressure.
With cooperative token, when lots of in-flight callers try to get the
cached item specified by key from memcache and get cache misses, only
the first few query requests (limited by by ``num_tokens``) will be able
get the cooperative tokens by creating or incrementing an internal
memcache key, and then those callers with tokens can send backend
requests to fetch data from backend servers and be able to set data into
memcache; all other cache miss requests without a token should wait for
cache filling to finish, instead of all querying the backend servers at
the same time.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Yan Xiao <yanxiao@nvidia.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Signed-off-by: Jianjian Huo <jhuo@nvidia.com>
Change-Id: I50ff92441c2f2c49b3034644aba59930e8a99589

Change-Id: Ib6bf702e38495e52e3b2f5ca95ed17c519018474
Signed-off-by: Clay Gerrard <clay.gerrard@gmail.com>

The disable_fallocate function provided in common.utils doesn't really
have a way to undo it - it's tested independently in test_utils.
It shouldn't be used on test_diskfile or else test_utils fallocate tests
will fail afterwards.
Change-Id: I6ffa97b39111ba25f85ba7cfde21440d975dc760
Signed-off-by: Clay Gerrard <clay.gerrard@gmail.com>

Change-Id: I0168ab113fdda60ed858ed0928356699399d4044
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Closes-Bug: #2122543
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: I8a2a96394734899ee48e1d9264bf3908968c51a8