openstack/swift - swift - OpenDev: Free Software Needs Free Tools

To me, this structure makes the call order more obvious. There was some
uneccessary defensiveness on the structure of the dict returned from the
SELECT query.
Drive-by: cleanup docstrings, there was some extra info about failures
and ContainerBrokers that was confusing me.
Related-Change: Ic7c2aa878caf039b29abb900b4f491130be3d8a8
Change-Id: I13e91abb09b2102dc52429df22fe47c73c6346aa
Signed-off-by: Clay Gerrard <clay.gerrard@gmail.com>

Move tests to base TestCase, currently they are only running against old
"broker w/o metadata" - but the tests and behavior should work on all
versions of the account schema
Drive-by: reword tests to make assertions stronger and behaviors more
obvious
Related-Change: Ic7c2aa878caf039b29abb900b4f491130be3d8a8
Change-Id: I59abd956ffa01bd41f29959ff3df89a3a20a00d4
Signed-off-by: Clay Gerrard <clay.gerrard@gmail.com>

iterate through not allowed methods and assert
Change-Id: Ia304709fc56d3e81bb1326b56a4b0d64ed698160
Signed-off-by: Tim Burke <tim.burke@gmail.com>

In addition to being in the pure unmolested ondisk format from the
source container; the manifest must also include the normally
protected X-Static-Large-Object metadata.
Change-Id: Ic6638e8258e9dec755f8d9630f0586bd3c9b4420
Related-Change: I8d503419b7996721a671ed6b2795224775a7d8c6
Signed-off-by: Tim Burke <tim.burke@gmail.com>

Change-Id: I99dbd9590ff39343422852e4154f98bc194d161d
Signed-off-by: Tim Burke <tim.burke@gmail.com>

Related-Change-Id: I38c11b7aae8c4112bb3d671fa96012ab0c44d5a2
Change-Id: Ibe5d206d2b96e174f849715fb13562ae0d2f5de2
Signed-off-by: Clay Gerrard <clay.gerrard@gmail.com>

Most of swift's timing configuration values should accept units in
seconds; make this explicit in the sample config for values that did not
already do so.
Related-Change-Id: I38c11b7aae8c4112bb3d671fa96012ab0c44d5a2
Change-Id: I5b25b7e830a31f03d11f371adf12289222222eb2
Signed-off-by: Clay Gerrard <clay.gerrard@gmail.com>

This patch replaces --force-commits with a --skip-commits flag in
swift-manage-shard-ranges to determine when to commit object updates.
Change-Id: I6de041f5c12dca2618d22d1271efe242b2f35258
Signed-off-by: Christian Ohanaja <cohanaja@nvidia.com>

The cost of memcache misses could be deadly. For example, when
updating shard range cache query miss, PUT requests would have to
query the backend to figure out which shard to upload the objects.
And when a lot of requests are sending to the backend at the same
time, this could easily overload the root containers and cause a
lot of 500/503 errors; and when proxy-servers receive responses of
all those 200 backend shard range queries, they could in turn try
to write the same shard range data into memcached servers at the
same time, and cause memcached to return OOM failures too.
We have seen cache misses frequently to updating shard range cache
in production, due to Memcached out-of-memory and cache evictions.
To cope with those kind of situations, a memcached based cooperative
token mechanism can be added into proxy-server to coalesce lots of
in-flight backend requests into a few: when updating shard range
cache misses, only the first few of requests will get global
cooperative tokens and then be able to fetch updating shard ranges
from backend container servers. And the following cache miss
requests will wait for cache filling to finish, instead of all
querying the backend container servers. This will prevent a flood
of backend requests to overload both container servers and memcached
servers.
Drive-by fix: when memcache is not available, object controller will
only need to retrieve a specific shard range from the container server
to send the update request to.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Yan Xiao <yanxiao@nvidia.com>
Co-Authored-By: Shreeya Deshpande <shreeyad@nvidia.com>
Signed-off-by: Jianjian Huo <jhuo@nvidia.com>
Change-Id: I38c11b7aae8c4112bb3d671fa96012ab0c44d5a2

Add a path property for AccountBroker and use lazy, resilient
_populate_instance_cache(). Use None attrs as flags, avoid broad
try/except in path, and retry if cache population fails.
Change-Id: Ic7c2aa878caf039b29abb900b4f491130be3d8a8
Signed-off-by: ashnair <ashnair@nvidia.com>

Memcached based cooperative token is a improved version of ghetto lock,
see the description of ghetto lock at here:
https://github.com/memcached/memcached/wiki/ProgrammingTricks
It's used to avoid the thundering herd situation which many caching
users face: given a cache item that is popular and difficult to
recreate, in the event of cache misses, users could end up with hundreds
(or thousands) of processes slamming the backend database at the same
time in an attempt to refill the same cache content. This thundering
herd problem not only often leads to unresponsive backend; and also
those writes into memcached cause premature cache eviction under memory
pressure.
With cooperative token, when lots of in-flight callers try to get the
cached item specified by key from memcache and get cache misses, only
the first few query requests (limited by by ``num_tokens``) will be able
get the cooperative tokens by creating or incrementing an internal
memcache key, and then those callers with tokens can send backend
requests to fetch data from backend servers and be able to set data into
memcache; all other cache miss requests without a token should wait for
cache filling to finish, instead of all querying the backend servers at
the same time.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Yan Xiao <yanxiao@nvidia.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Signed-off-by: Jianjian Huo <jhuo@nvidia.com>
Change-Id: I50ff92441c2f2c49b3034644aba59930e8a99589

Change-Id: Id2ab5c00182516a744fd9e8b8e89f7232a433222
Signed-off-by: Christian Ohanaja <cohanaja@nvidia.com>

Change-Id: Ib6bf702e38495e52e3b2f5ca95ed17c519018474
Signed-off-by: Clay Gerrard <clay.gerrard@gmail.com>

The disable_fallocate function provided in common.utils doesn't really
have a way to undo it - it's tested independently in test_utils.
It shouldn't be used on test_diskfile or else test_utils fallocate tests
will fail afterwards.
Change-Id: I6ffa97b39111ba25f85ba7cfde21440d975dc760
Signed-off-by: Clay Gerrard <clay.gerrard@gmail.com>

Change-Id: I0168ab113fdda60ed858ed0928356699399d4044
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Verified every changed link works by
building and testing manually
Change-Id: I4bb6cc238d4e567e3edc6c15a58d4a5f9a41e273
Signed-off-by: Christian Ohanaja <cohanaja@nvidia.com>

Add file to the reno documentation build to show release notes for
stable/2025.2.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2025.2.
Sem-Ver: feature
Change-Id: I3ab48efc8208b791bfdf5ac24d098b8e236d7031
Signed-off-by: OpenStack Release Bot <infra-root@openstack.org>
Generated-By: openstack/project-config:roles/copy-release-tools-scripts/files/release-tools/add_release_note_page.sh

Closes-Bug: #2122543
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: I8a2a96394734899ee48e1d9264bf3908968c51a8

Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: I9c86383ed3d35657a7e88fa9cdc6a94559e5ca37

Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: I2e2a4e2c448319e6531372ae06ab81eb58edc57e

Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: I26be3752b8c67b7a6a0a9c75571a44f3827cbb90

- Drop py36 versions from py3-constraints.txt
- Remove py36 tox environment
Forgot to do this when we dropped py36 support.
Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: I0233e1dd036b9a420c815fec3c9632d2967b934e

We're starting to see projects drop py39 support; seems as good a time
as any for an update.
Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: I481efbd2627e517edf49f3025b3399a86d1b4f3e

Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: I6fb806d299fc30f6ceaeba78cf3a810298e94f26

Added the new field to be able to log the access key during the
s3api calls, while reserving the field to be filled with auth relevant
information in case of other middlewares. Added respective code to
the tempauth and keystone middlewares.
Since s3api creates a copy of the environ dict for the downstream
request object when translating the s3req.to_swift_req the environ
dict that is seen/modifed in other mw module is not the same instance
seen in proxy-logging - using mutable objects get transfered into the
swift_req.environ.
Change the assert in test_proxy_logging from "the last field" to
the index 21 in the interests of maintainability.
Also added some regression tests for object, bucket and s3 v4 apis and
updated the documentation with the details about the new field.
Signed-off-by: Vitaly Bordyug <vbordug@gmail.com>
Change-Id: I0ce4e92458e2b05a4848cc7675604c1aa2b64d64

Closes-Bug: #2120783
Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: Id175329d717bb4959def8b43765fec668aa26696

S3 stopped returning DisplayNamme in the Owner field of object
listings [1], so the tests need to stop asserting that it is present.
Further work is needed to drop DisplayName from the Swift s3api
responses [2].
[1] https://docs.aws.amazon.com/AmazonS3/latest/API/API_Owner.html
[2] https://bugs.launchpad.net/swift/+bug/2120622
Change-Id: Ia915a65313394910c74ae826c912b5549e833a7b
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

The RuntimeError gives more of a hint than the TypeError, but we haven't
really solved the issue. If there's a busted pyeclib install, it's safe
to assume it won't have ISA-L alongside it; just proceed assuming it's
not available.
Closes-Bug: #2120591
Related-Change: I64a85eb739fb72efe41f1ee829e463167246b793
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: I2791566b208327b1fb536fb56a363337ab3f3941

In order to continue testing on py37, start using pyenv to install it.
Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: Ida8be4477116d3b44e3d7b470a183d2298e2d3d5

This reverts commit 33b89492eb.
Reason for revert: Some packaging workflows still need this.
Closes-Bug: #2120590
Signed-off-by: Matthew Oliver <matt@oliver.net.au>
Change-Id: I5cfb8e6412adea30a1a21ce20f6a72c30dfbfac3

The related change added a py3.13 job tho the check pipeline but not
the gate. This patch adds it to the gate too.
Change-Id: Ie7e676e7bb2e5f975b560c083432ce6814d38252
Related-Change: Ib797f73134c8fa5ef0dc66aeddf06adfa50ed7d4
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>