openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Newer versions of python expect to be able to say `with self.lock:`
down in logging.
See https://github.com/python/cpython/commit/74723e11
Change-Id: I30305566d12a1b8be4c8bde4b416b798322a1385

It is currently possible to configure "X-Container-Meta-Quota-Bytes"
and "X-Container-Meta-Quota-Count" on containers, as well as
"X-Account-Meta-Quota-Bytes" on accounts. However, there is no way
to configure an account quota limit for the number of files or
"quota-count". This limitation could potentially allow a user to
exhaust filesystem inodes.
This change introduces the "X-Account-Meta-Quota-Count" header,
allowing users with the ResellerAdmin role to add a quota-count
limit on accounts. The middleware will enforce this limit similarly
to the existing quota-byte limit.
Co-authored-by: Azmain Adib <adib1905@gmail.com>
Co-authored-by: Daanish Khan <daanish1337@gmail.com>
Co-authored-by: Mohamed Hassaneen <mohammedashoor89@gmail.com>
Co-authored-by: Nada El-Mestkawy <nadamaged05@gmail.com>
Co-authored-by: Tra Bui <trabui.0517@gmail.com>
Change-Id: I1d11b2974de8649a111f2462a8fef51039e35d7f

When the proxy passes the container-update headers to the object server
now include the db_state, which it already had in hand. This will be
written to async_pending and allow the object-updater to know more about
a container rather then just relying on container_path attribute.
This patch also cleans up the PUT, POST and DELETE _get_update_target
paths refactoring the call into _backend_requests, only used by these
methods, so it only happens once.
Change-Id: Ie665e5c656c7fb27b45ee7427fe4b07ad466e3e2

The ssl.wrap_socket method has been removed in 3.12.
SSLContext.wrap_socket should now be used.
Change-Id: I6119e054289eac263ff5448d7d118209f98678d9

The proxy may use a sub-request to fetch a container listing response,
for example for updating shard ranges. If it then fails to parse the
response, it should use the sub-request path when logging a warning or
error. Before, the proxy would use the original request path which may
have been to an object.
Change-Id: Id904f4cc0f911f9e6e53d4b3ad7f98aacd2b335d

Use same assertion pattern as introduced in the Related-Change and
include the host and port in the assertion.
Change-Id: I7cccf3b8cdfe157c27fb8e0b1f432a649970a865
Related-Change: I9a016de1010dd17e91bc85877413b34d2c3287c7

With labeled metrics on the horizon, I wanted some more end-to-end
tests of stats emission.
Change-Id: I9a016de1010dd17e91bc85877413b34d2c3287c7

It takes a config dict and some caller-specific options, similar to
get_logger. Use this in get_logger, so our logging module doesn't
need to know anything about statsd config options.
Co-Authored-By: yanxiao@nvidia.com
Change-Id: I5ae2cc5c257fb8d7eab885977d9d9cf602224ec7

There's a variety of reasons why s3api may return a 4xx or 5xx
response. It's useful to get some more detail in the log message,
particularly when the request is a HEAD which won't get any detail in
the response body.
Change-Id: I0a499dc5c50bb534a23adcbfe7c3822d8954b0e0

The size in bytes from object metadata of expiring objects are stored in
expirey queue entries under the content_type field.
The x-content-type-timestamp take from object metadata is provided along
with the x-content-type update so the container replicator resolves the
latest content-type and ensures eventual consistency.
UpgradeImpact: During rolling upgrades you should expect expirer queue
entries to continue lacking swift_expirer_bytes= annotations until ALL
object servers replicas have been upgraded to new code.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: Ie4b25f1bd16def4069878983049b83de06f68e54

There was an abandoned change that made reference to a RecussionError
when running a probe test that imported boto3 that had something to do
with eventlet, ssl and a transitive dependency on requests-mock, but the
fix that actually got merged seemed to depend on another change to
tox.ini that disables request-mock when we run pytest.
Either way, we already import from boto3 at the top of probe tests and
it's in test-requirements; so we require it to be installed even if you
don't have s3api in your pipeline.
Related-Change: I789b257635c031ac0cb6e4b5980f741e0cb5244d
Related-Change: I2793e335a08ad373c49cbbe6759d4e97cc420867
Related-Change: If14e4d2c1af2efcbc99e9b6fe10973a7eb94d589
Change-Id: Id2662bfc5ef2f21f901f1c98e6389c4cb01818a2

Change-Id: Ie5d8b555489d28c3b901e5bbebdcecbde7bb3367

Change-Id: I9413c72f41465fb8026848f71ec3b39fa990c3b7

Drive-by: fix ValueError message for non_negative_int
Change-Id: I06508279d59fa57296dd85548f271a7812aeb45f

We've been working toward separating our logger from our statsd client.
This is generally a good idea; it's always been a little weird to have
our special-case loggers that would allow you to *also* increment some
counters.
The end goal is to take a bunch of places that look like
 logger = utils.get_logger(conf)
 ...
 logger.info(...)
 logger.increment(...)
and turn them into something more like
 logger = logs.get_adapted_logger(conf)
 stats = statsd_client.get_statsd_client(conf, logger=logger)
 ...
 logger.info(...)
 stats.increment(...)
Take a lesson from logging: callers don't need to know whether the
log_level is high enough that their message will be logged, or even
whether logging is enabled at all. Code wanting to emit stats shouldn't
need to know whether statsd collection has been configured, either.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I6eb5b27a387cc2b7310ee11cc49d38fd2b6cbab8

Even if stdlib socket is probably patched by the time StatsdClient
creates a socket, we want to import the green socket module explicitly
for better testing.
Move test_statsd.py to test_statsd_client.py so it matches the naming
convention of the rest of our test files.
Fix some patching of utils in test_statsd_client to patch
statsd_client.
Rename some vars in test_statsd_client that shadowed the statsd_client
module name.
Move some utils tests out of test_statsd_client and back into
test_utils.
Related-Change: I4b5b12a3b0288b696a39903264741bc862a94ad7
Change-Id: I3de22b7f15dd386fa9c873587782f0dfc4c42a27

Related-Change: I9924ff3b8d7d246631fe61b916823e028e2c01f2
Change-Id: I2a4581e408e3012a44bfb0cd58563b82040e23fc

Adds ClosingMapper class which is like map() but closes the
iterable.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: Idd0ac21b365a138b065f01d05a257af62ea88177

Change-Id: I4b5b12a3b0288b696a39903264741bc862a94ad7

The boto library was last updated two years ago and has
rusted to the point that it's unusable on py312 -- see
https://github.com/boto/boto/issues/3951
We should transition all of these tests to boto3 equivalents,
but this should help out in the meantime.
Related-Bug: #1557260
Related-Bug: #2063367
Change-Id: If95f45371f352c6a2d16be1a3e1b64e265bccfb4

This shouldn't impact real servers, as those processes were about to
wrap up anyway. It *can* cause some confusing behaviors in tests,
though.
Change-Id: Ifd8a64efcd3fc983596ba7cd9fe28eb9663c93d6

Change-Id: I68974338f8e0284ed77960048a83f72855b93348

Change-Id: I33e71f6c201bb4f2cf3481afd40cf489eb1fcd1f

- Move statsd client into it's own module
- Move all logging functions into their own module
- Move all config functions into their own module
- Move all helper functions into their own module
Partial-Bug: #2015274
Change-Id: Ic4b5005e3efffa8dba17d91a41e46d5c68533f9a

Related-Change: I106103438c4162a561486ac73a09436e998ae1f0
Change-Id: I95606aed5a5bd70424fbc51dd965f238fa7f064f

If the global configuration option 'enable_open_expired' is set
to true in the config, then the client will be able to make a
request with the header 'x-open-expired' set to true in order
to access an object that has expired, provided it is in its
grace period. If this config flag is set to false, the client
will not be able to access any expired objects, even with the
header, which is the default behavior unless the flag is set.
When a client sets a 'x-open-expired' header to a true value for a
GET/HEAD/POST request the proxy will forward x-backend-open-expired to
storage server. The storage server will allow clients that set
x-backend-open-expired to open and read an object that has not yet
been reaped by the object-expirer, even after the x-delete-at time
has passed.
The header is always ignored when used with temporary URLs.
Co-Authored-By: Anish Kachinthaya <akachinthaya@nvidia.com>
Related-Change: I106103438c4162a561486ac73a09436e998ae1f0
Change-Id: Ibe7dde0e3bf587d77e14808b169c02f8fb3dddb3

The object expirer can be configured to delay the reaping of
objects from disk after their expiration time using account
and container level delay_reaping values. The delay_reaping
value of accounts and containers in seconds is configured in
the object server config. The object expirer references these
configured values to only reap objects from specified accounts
and containers after their corresponding delays.
The goal of the delay_reaping feature is to prevent accidental or
premature data loss if an object marked for deletion with the
'x-delete-at' feature should not be reaped immediately, for
whatever reason.
Configuring the delay_reaping value at a granular account and
container level is beneficial for being able to keep storage
capacity consumption in control while maintaining a desired
data recovery window.
This patch also adds a sample configuration, documentation, and
tests for bad configurations and grace period functionality.
Co-Authored-By: Anish Kachinthaya <akachinthaya@nvidia.com>
Change-Id: I106103438c4162a561486ac73a09436e998ae1f0

Add some test assertions to cover the first-byte timing metrics
introduced in the related change.
Add ttfb param to log_request docstring.
Change-Id: I530652dd672d7d4e5eac351ccbad318773414f7d
Related-Change: I1611e34846e586703e9d3709fa64e8df41f2d685

Change-Id: I526bbcc59c9eb5923c3784d5d06bc38998cb48db

Change-Id: I651ddd40e9115a56727096d4a3aa84589146308f

Previously, we relied on raising a swob.HTTPException from within a
wrapped wsgi.input and just hoping that it either
- comes uncaught all the way back through the pipeline, or at least
- gets translated back to a 422 response.
This was brittle, however; if any middleware between s3api and the
proxy took s3api's approach of turning unexpected responses into 500s,
for example, it all breaks down.
Take a cue from eventlet's Timeout and create a new s3api-specific
BaseException (which does *not* inherit from Exception) to improve
our ability to cut through all the layers.
Change-Id: I9924ff3b8d7d246631fe61b916823e028e2c01f2