openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Personally I'm not a big fan of how we arrange logs for SAIO,
but it is a historic standard. The reconciler has to conform.
Change-Id: I45a25ff406b31b6b1b403e213554aaabfebc6eb5

For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html
Change-Id: I7b9907645d7357824e197fa213f0ec68283ee72b

The client_chunk_size attribute was introduced into GetOrHeadHandler
for EC support [1]. It was only ever not None for an
ECObjectController. The ECObjectController stopped using
GetOrHeadHandler for Object GET when the ECFragGetter class was
introduced [2], but the EC specific code was not expunged from
GetOrHeadHandler. In [3] the ECFragGetter client_chunk_size was renamed
to fragment_size to better reflect what it represented.
The skip_bytes attribute was similarly introduced for EC support. It
is only ever non-zero if client_chunk_size is an int. For EC,
skip_bytes is used to undo the effect of expanding the backend
range(s) to fetch whole fragments: the range(s) of decoded bytes
returned to the client may need to be narrower than the backend
ranges. There is no equivalent requirement for replicated GETs.
The elimination of client_chunk_size and skip_bytes simplifies the
yielding of chunks from the GetOrHeadHandler response iter.
Related-Change:
[1] I9c13c03616489f8eab7dcd7c5f21237ed4cb6fd2
[2] I0dc5644a84ededee753e449e053e6b1786fdcf32
[3] Ie1efaab3bd0510275d534b5c023cb73c98bec90d
Change-Id: I31ed36d32682469e3c5ca8bf9a2b383568d63c72

Change-Id: I5e75a359660315a34ec36f5cb748808257f4185a

The Related-Change extracted some attributes of the ShardRange class
into a new Namespace superclass. This patch adds a docstring for
Namespace.
Change-Id: I5a79c4a6da6e62698403bb0a5ef566355b5c850e
Related-Change: If98af569f99aa1ac79b9485ce9028fdd8d22576b

Within ContainerBroker, at various places, for example get_db_state()
and sharding_initiated(), they query the number of shard ranges of
this container by pulling out all shard ranges from shard range table,
instantiating ShardRange objects and then counting how many they are.
Those operations are very expensive, and causing HEAD/GET into large
containers to be very slow.
Instead, this patch only checks if there is any qualified shard range
exists in the shard table with optimized SQL query, which can be very
fast. On a container server setup which serves a container with ~12000
shard ranges, this patch alone improves HTTP HEAD request rate by ~10X,
and improves HTTP GET request rate by ~2X; and together with other
optimizations (patch 888310 & 888575) targeting to fix similar problems,
strong synergistic effects are seen to bring total ~22X improvement to
HTTP HEAD and ~27X to HTTP GET request rates.
Change-Id: I01fd4f3e395c8846280f44e17a56935fc6210444

The 'log_route' argument of utils.get_logger() determines which global
Logger instance is wrapped by the returned LogAdapter. Most middlewares
(s3api being the exception) explicity set 'log_route' to equal the
middleware 'brief' name e.g. 'bulk', 'tempauth' etc. However, the
s3api middleware sets 'log_route' to be the config 'log_name', if that
key is found in config.
When a proxy pipeline is instantiated via wsgi.run_wsgi(), all
middlewares and the proxy app are passed a default conf with
'"log_name": "proxy-server"'. As a result, the s3api middleware calls
get_logger() with log_route='proxy-server' and its LogAdapter
therefore shares the same Logger instance used by proxy-server app
(and any other middleware that similarly fails to explicitly
differentiate 'log_route)'.
Each Logger instance has a StatsdClient instance bound to it by
get_logger(). The Related-Change added statsd metrics to the s3api
middleware and sets 's3api' as the 'statsd_tail_prefix' when calling
get_logger(). This had the unintended effect of replacing the shared
Logger instance's StatsdClient with one that has prefix 's3api', such
that stats emitted by the proxy app (e.g. memcache shard range
hit/miss stats) would be erroneously prefixed with 's3api'.
This patch modifies the s3api middleware logger instantiation to
explictly set log_route='s3api', so that the s3api middleware
LogAdapter now wraps a unique global Logger instance, with a unique
StatsdClient instance bound to it.
The 'server' attribute of the middleware's LogAdapter, which may be
included in log lines by the "%(server)s" format element, is not
affected by this change. Its value is derived from the config
'log_name' or the 'name' argument passed to get_logger().
Change-Id: Ia89485bae8f92f4f3d9f5375cab8ff08f70a11a7
Related-Change: I4976b3ee24e4ec498c66359f391813261d42c495

Change-Id: Icdc74856f7beb318bfd84b39b5776d0d531a74b9

Prevent encoding corruption in client's metadata during ssync
Closes-Bug: #2020667
Change-Id: I0ea464bcda16678997865667287aa11ea89cdcde

Currently, SLO manifest files will be evicted from page cache
after reading it, which cause hard drives very busy when user
requests a lot of parallel byte range GETs for a particular
SLO object.
This patch will add a new config 'keep_cache_slo_manifest', and
try keeping the manifest files in page cache by not evicting them
after reading if config settings allow so.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I557bd01643375d7ad68c3031430899b85908a54f

Change-Id: I035ef2dcc0d0e09337bd2e742baeff1fb62bf018

It costs us an extra HEAD sometimes, but those at least get cached
some. Seems better than doing real GETs and going out to handoffs
every time when versioning isn't enabled.
Change-Id: Ic1b3a8c6c9b1aaead25070e49f514785c2d52c98

Currently we make no effort in formatting the meta and sysmeta
inside a db when we use the info (swift-{container,account}-info
tools. This patch properly formats them so they are easier to parse by human eyes.
Change-Id: I5c3d260d677c61213c42662c6641207d9b7f026a

Having a class named TestSource that is not a TestCase causes pytest
warnings:
 "PytestCollectionWarning: cannot collect test class 'TestSource'"
Change-Id: Ie63fdd8f3febe441963bc5755175878433a8c4c7
Related-Change: Ie8e0560f90207eede1992c8665e4d8571a505580

Users sometimes ask why their request received a 403
response. Sometimes s3api will include a reason in the response body,
but client code may not make this visible to the user. To provide some
other insights, this patch adds statsd metrics when error responses,
such as 403, are returned from the s3api middleware.
The new metrics have the form:
 s3api.<status_int>.<error_class>[.reason]
For example:
s3api.403.SignatureDoesNotMatch
s3api.403.RequestTimeTooSkewed
s3api.403.AccessDenied.invalid_date
s3api.400.InvalidBucketName
Change-Id: I4976b3ee24e4ec498c66359f391813261d42c495

The proxy has a class of "recoverable" timeouts; when they trip, it
tries to resume from another node, with a range offset that accounts
for the bytes already served.
If we set the X-Backend-Ignore-Range-If-Metadata-Present header (like
we do in SLO), it may cause the object servers to return the *whole*
object. When that happens on the initial GET, the proxy is smart enough
to still track the appropriate range to fetch when resuming -- but then
it would send the ignore-range header so the new object-server would
send the whole object.
This would cause problems in one of two ways:
 For replicated policies, too many bytes would be yielded out of
 the proxy app. proxy-logging would notice, truncate the response,
 and log a warning about it.
 For EC policies, at least one frag iter would have the wrong
 length, causing a decode error from pyeclib.
Now, pop off that header, if present.
Change-Id: I31f622f6ebfa255d4c52089ca182cbf665c40a30

Change-Id: Iadc54111e79910dd1809e21facba81153ca61822

Replace method for determine major:minor of block device
because stat can't detect major:minor in some cases.
Change-Id: Idcc7cd7a41e225d1052c03ba846dff02851758f8

For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html
Change-Id: I2a5f3e3d841d52d013b481118f56f7a1f6ab3f98

In some cases, when resuming a get using a new source node, the new
source node would be error limited rather than the old node. One case
was previously fixed [1], but other erroneous cases remain.
This patch adds a helper method to unify handling of all cases when an
old node should be error limited, and ensures that the old rather than
new node is limited.
[1] Related-Change: I02f9c0ffd879423211b87f7ac5acde762933f47d
Change-Id: Ibd405b79820cedaf9cb02906b8c13c8d34f4d7ec

The GetOrHeadHandler class in base.py and the ECFragGetter
class in obj.py have some methods that are very similar.
The two existing classes do also have unique code so they
will remain, but as subclasses of the common base class
called GetterBase.
Change-Id: I893c5fcb6b4f8a7dda351169f5f6b37375a34817

Related-Change: Ie2a8e4eced6688e5a98aa37c3c7b0c13fd2ddeee
Change-Id: Ic19964f1fc4e99f586364aefa1033800dd9bf143

RemoteDisconnected from both BadStatusLine and ConnectionResetError
(which in turn eventually inherits from OSError). We want to make
sure it gets handled as a BadStatusLine, as it doesn't get its errno
set and would otherwise get the default traceback handling.
Change-Id: I0fb1f764722d73db6d3b79acc128f37f51499d35

shutil.rmtree started using os.rmdir in a way that made our tests fail.
Change-Id: I59add10d886134af86e1bddec2ba39931c1213cb

Jammy only offers a py311 RC, so include the __slots__ hack to avoid
the segfault from https://github.com/python/cpython/issues/99886.
Fix up a test to work with the slotted connection.
Change-Id: I0e928bcb3810e391297300f4949024db3cf87d05

py311 made EntryPoints immutable. Subclass instead.
Closes-Bug: #2009228
Change-Id: I90198b1bb6b18b752e0fdc4d4d920914ea449413