openstack/swift - swift - OpenDev: Free Software Needs Free Tools

- SHA256 mismatches should trip XAmzContentSHA256Mismatch errors,
 not BadDigest. This should include ClientComputedContentSHA256 and
 S3ComputedContentSHA256 elements.
- BadDigest responses should include ExpectedDigest elements.
- Fix a typo in InvalidDigest error message.
- Requests with a v4 authorization header require a sha256 header,
 rejecting with InvalidRequest on failure (and pretty darn early!).
- Requests with a v4 authorization header perform a
 looks-like-a-valid-sha256 check, rejecting with InvalidArgument
 on failure.
- Invalid SHA256 should take precedence over invalid MD5.
- v2-signed requests can still raise XAmzContentSHA256Mismatch errors
 (though they *don't* do the looks-like-a-valid-sha256 check).
- If provided, SHA256 should be used in calculating canonical request
 for v4 pre-signed URLs.
Change-Id: I06c2a16126886bab8807d704294b9809844be086

My home server has more strict controls on /tmp as it run selinux etc.
When running unittests and the default log_filename_prefix deep under
/tmp gets permission denied. It would be better to override this setting
in the tests with a good known tmp location
Change-Id: I6c95ca3a0045a8f268802c6abb633bdfb0e56b73

Change-Id: I128a64c6126014774bf6218502e6bfdea63b783f
Partial-Bug: #1674543 

Those were so unwieldy as to be uneditable before.
Change-Id: Ic9f4a0ea6b8e18e1624c516890ab69884a299773

Change-Id: If5793d1b18cf8f35b8f98206f029f6cc6ca0cf1e

Previously, when handling an object POST, if the proxy got a 202 and
two connection timeouts from the 3 primary backend object servers, it
then went to two handoffs which return 404. The proxy would consider
this to be a quorum of 404s and return the client a 404 response. This
is alarming for the client.
With this patch, the proxy will only treat a 404 response from a
handoff as authoritative if it has an x-backend-timestamp header
(i.e. there's a tombstone on the handoff). POST responses never have
an x-backend-timestamp header so in the scenario described above the
proxy will return a 503.
The Related-Change previously made a similar fix such that 404s from
handoffs are already non-authoritative for object GETs and HEADs
unless they have an x-backend-timestamp header.
Related-Change: Ia832e9bab13167948f01bc50aa8a61974ce189fb
Cloeses-Bug: #2077743
Change-Id: I96f28ab0b2b5f9374c399e8905ee240e7b093f8b

Add unit tests for the proxy object controller to cover object DELETE
and POST scenarios.
Change-Id: I625a4cf03ee9d4a270d60fa2dc9795b36bb36bf1

Sharding involves both container DB state transitions and ShardRange
state transitions. To avoid confusion, always use db_state as the var
name when referring to the container DB state value.
Change-Id: Iaaf494fd4e02017005cb3811b673f967bd6b5e1d

Some unit tests for the proxy object GET and HEAD path were setup with
insufficient mock responses. The mock connection would raise
StopIteration exceptions once the mock responses were exhausted, which
the object controller would handle as if they were error responses,
potentially distorting the test scenario.
This patch makes the set_http_connect context manager check for
unexpected requests and raise an AssertionError if any are found.
Change-Id: I47774396d9d0a78ebceea6c628c9412b3ad67a11

Running unittests on my home server, even though is a beast, would be
painfully slow. Running them on my laptop ran much faster. I tried
everything and couldn't figure it out.
Chris, co-author, dug in and after we ran out of options he fired up his
packet capture and realised most tests were issuing DNS queries for
`host.`. On my home server, it runs ipv6 dual stack, so these calls
would need to timeout before a test can continue.
On finding this `host.` Ben, another co-author, dug into the code and
found the only reference we have to `host` is in FakeStatsdClient.
Sure enough as I dug a little further it turns out our FakeStatsdClient
used to override a StatsdClient function `_determine_sock_family(self,
host, port)` to stop actually creating a real socket.
However, at some point the StatsdClient was refactored and that method
was renamed and changed. Which leads to a DNS lookup every time we
create a debug_logger as it brings up the socket.
As you can imagine, this happens alot!
This patch overrides the new function that handles to socket creation
`_set_sock_family_and_target(self, host, port)`. Which eliminates the
DNS call all together.
Co-Authored-By: Ben Formosa <bformosa@nvidia.com>
Co-Authored-By: Chris Smart <csmart@nvidia.com>
Change-Id: Ie393075f79447627714692e3f01bb53e967a71e8

Change-Id: Ie73988edf6be0e38d9004bee04ff46c906a759ff

Some tests in test_account_quotas.py registered a response with
FakeSwift during setUp and then subsequently mutated the response's
headers dict. Worse, some tests then mutate the headers again and
re-use the same response.
This relies on the FakeSwift implementation not copying the original
headers when the response is registered, which may not remain the case
(as the author discovered with the Related-Change). It's also an
unnecessary shortcut.
Change-Id: I3217c17936d0c11b03de4a4a172bb2fb0a2be734
Related-Change: I84604a7ea049850ad4d0f0cea2096ab8a98dfb4a

Change-Id: Ibd111773a496e8e3b1fc1577f40aa69f4328139b

Drive-by: simplify patching of test expirer instance.
Drive-by: fix comment spelling.
Related-Change: Ib151c498ba325f39570e963e5b7948080ffcd3d6
Change-Id: Ia0041c4eedc33a2675890d52cbbc33b9190f1665

Four new metrics have been added into task iteration process:
 'tasks.parse_errors': count of errors when parsing the task object
 'tasks.skipped': count of task objects skipped because it doesn't
 belong to this expirer.
 'tasks.delayed': count of objects is still within the delay
 'tasks.assigned': count of assigned objects to this expirer
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: Ib151c498ba325f39570e963e5b7948080ffcd3d6

Cleans up a deprecation warning:
 DeprecationWarning: Passing more than 1 positional argument to
 _sqlite3.Connection() is deprecated. Parameters 'timeout',
 'detect_types', 'isolation_level', 'check_same_thread', 'factory',
 'cached_statements' and 'uri' will become keyword-only parameters
 in Python 3.15.
Change-Id: I2b11487f8174e36b024c503d7e78de1904e0a281

Newer versions of python expect to be able to say `with self.lock:`
down in logging.
See https://github.com/python/cpython/commit/74723e11
Change-Id: I30305566d12a1b8be4c8bde4b416b798322a1385

The container-info cli already supports the --sync/-s option to view
incoming and outgoing syncs from the db. Account works exactly the same
way. And hey be might want to interrogate who an account has been
syncing with. So it should have the same option.
This patch adds the same option to account_main parser:
 $ swift-account-info <path_to_account> -s
 Path: /a2
 Account: a2
 Deleted: False
 Account Hash: c1e207026da71a9237c5d9e206b9ec59
 Metadata:
 Created at: 2024年07月24日T06:08:28.760490 (1721801308.76049)
 Put Timestamp: 2024年07月24日T06:08:28.743650 (1721801308.74365)
 Delete Timestamp: 1970年01月01日T00:00:00.000000 (0)
 Status Timestamp: 2024年07月24日T06:08:28.743650 (1721801308.74365)
 Container Count: 10
 Object Count: 100
 Bytes Used: 880
 Chexor: 0673fe17a762662d74e8d3ad93e0785b
 UUID: 705ac602-5e85-49ef-92a4-3082d40cad4f-sdb3
 ...
 Incoming Syncs:
 Sync Point	Remote ID 	Updated At
 10 	285bbaae-90da-409a-9533-f0d113764e44-sdb4	2024年07月24日T06:10:34.000000 (1721801434)
 11 	80c45862-dcd3-4ffe-9659-91851276d244-sdb2	2024年07月24日T06:10:35.000000 (1721801435)
 Outgoing Syncs:
 Sync Point	Remote ID	Updated At
 ...
Change-Id: I1aa96e97281421c69cb75ea24ce427c20c4c615e

- swift-account-audit
 - swift-config
 - swift-drive-audit
 - swift-get-nodes
 - swift-reconciler-enqueue
 - swift-oldies
 - swift-orphans
Related-Change: Ifcc8138e7b55d5b82bea0d411ec6bfcca2c77c83
Change-Id: Ie4a252ec51d23caa9e90e6c2f772847d0d71b1c8

Related-Change: Ifcc8138e7b55d5b82bea0d411ec6bfcca2c77c83
Change-Id: I2718d8b96193d13f913824fc1091d72737d755ea

Unfortunately, kotori had to be switched to a whitelist,
so it's not appropriate for a public address list anymore.
Change-Id: I86f10fcf4fd0bc5902c3dad49c6f3019fa159b64

Recent OSes are using ISO format for their kernel logs.
Closes-Bug: #2072609
Change-Id: I92fce513d06d8b0875dabf9e9a1b2c5a3a79d9b5

Prior to the Related-Change, non_negative_int would raise a ValueError
if it was passed a positive float. It now casts the float to an int,
which is consistent with the docstring.
This patch adds test assertions to verify this behavior, and similar
behavior of config_positive_int_value.
Change-Id: I5d62e14c228544af634190f16321ee97a8c4211c
Related-Change: I06508279d59fa57296dd85548f271a7812aeb45f

Add unit tests to verify the precedence of access_log_ and log_
prefixes to options.
Add pointers from proxy_logging sections in other sample config files
to the proxy-server.conf-sample file.
Change-Id: Id18176d3790fd187e304f0e33e3f74a94dc5305c

Related-Change: Ifcc8138e7b55d5b82bea0d411ec6bfcca2c77c83
Change-Id: I7f479359f94f102414f30e8aac87f377d10d34ee

I *think* swift.common.manager is a reasonable place for it?
Related-Change: Ifcc8138e7b55d5b82bea0d411ec6bfcca2c77c83
Change-Id: I48a345eedbf2369d481d18c0e2db37845673b647

See https://review.opendev.org/c/openstack/swift/+/918365 for motivation.
A handful of [files]scripts entries still remain, but they're more
involved to change over. These ones were all fairly mechanical to fix.
Related-Change: Ifcc8138e7b55d5b82bea0d411ec6bfcca2c77c83
Change-Id: Ia43d7cd3921bc6c9ff0cee3100ef5f486fd3edcb

It is currently possible to configure "X-Container-Meta-Quota-Bytes"
and "X-Container-Meta-Quota-Count" on containers, as well as
"X-Account-Meta-Quota-Bytes" on accounts. However, there is no way
to configure an account quota limit for the number of files or
"quota-count". This limitation could potentially allow a user to
exhaust filesystem inodes.
This change introduces the "X-Account-Meta-Quota-Count" header,
allowing users with the ResellerAdmin role to add a quota-count
limit on accounts. The middleware will enforce this limit similarly
to the existing quota-byte limit.
Co-authored-by: Azmain Adib <adib1905@gmail.com>
Co-authored-by: Daanish Khan <daanish1337@gmail.com>
Co-authored-by: Mohamed Hassaneen <mohammedashoor89@gmail.com>
Co-authored-by: Nada El-Mestkawy <nadamaged05@gmail.com>
Co-authored-by: Tra Bui <trabui.0517@gmail.com>
Change-Id: I1d11b2974de8649a111f2462a8fef51039e35d7f