openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Apparently errno.EUCLEAN is not available on OS X, which can complicate
running unit tests.
Change-Id: Iaa3d7756949b4a67d4afe8a53b242ed9f41e9374

When Expirer is iterating task containers and doing listings, it's
possible one of the container nodes hosting the task container may
become overloaded (or have a really low backend ratelimit set).
Object-expirer should expect UnexpectedResponse and continue to try
and list the task objects in the next container.
And if the task container doesn't exist, expirer should not try to
delete the non-existent containers, before continue to work on
the next container.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: Id1966fa22725a02471e2d7c5a42fb243b1cfcf6a

Previously the object-expirer would not respect an internal_client_conf_path option if its config was loaded from a "legacy" config filepath (e.g. **/object-expirer.conf). Legacy object-expirer config expected config sections for an internal-client to be included in the same file. However, "modern" object-expirer config files (e.g. **/object-server.conf) expect internal-client config to be loaded from a separate file specified by internal_client_conf_path and defaulting to 'internal_client_conf_path'.
"Legacy" expirer config is still used in production clusters but it is useful to use a shared internal-client config. This patch therefore makes the internal_client_conf_path option always be respected regardless of the path of the object-expirer conf file. If 'internal_client_conf_path' is not specified, "modern" config will continue to use the default '/etc/swift/internal-client.conf', but "legacy" config will default to the path of the expirer conf file (i.e. the same as the previous behavior).
Related-Change: Ib21568f9b9d8547da87a99d65ae73a550e9c3230
Change-Id: I24ec702cd2ed074ca9df084cefc896418cece394

py2 http responses don't have a 'headers' attribute, so don't try to
use it.
Unfortunately the unit test infrastructure's FakeConn class does
have a 'headers' attribute (whose value doesn't match the result of
calling its 'getheaders()' method!), so this bug was not caught by
tests in the Related-Change.
Related-Change: I96f28ab0b2b5f9374c399e8905ee240e7b093f8b
Change-Id: I2cd820280b8c69cafc5730183903c9d379d8dde5

Previously the in-process functional test setup attempted to insert
etag-quoter in the proxy pipeline by replacing the substring 'cache
listing_formats' with 'cache etag-quoter listing_formats'. However,
the pipeline had already been modified to have 'cache domain-remap
listing_formats', so the etag-quoter replacement failed silently.
This patch modifies the etag-quoter replacement to match just
'cache'. Some helper functions are also introduced to make the
pipeline modification more uniform and to consistently verify the
intended modifications have been achieved, or otherwise raise an
exception.
Change-Id: I640837a4da985bcbe48f9ae5eeb56385b11034eb

A print statement, presumably for debugging, crept in with the
Related-Change.
Change-Id: I0cccb960b51446a3aa58be95c8c9d06cc6b6eada
Related-Change: I69974cc8a39731c980b54137b799a36b2e63a44a

Some s3api functional tests were setting boto logging level to DEBUG,
which results in a very large amount of logging output from the test
runner, including large request bodies. This makes it extremely hard
to inspect test output.
This patch makes the quiet_boto_logging context manager revert the
logger level to its original value rather than indiscriminately
leaving it set to DEBUG.
Change-Id: I1dd9603adf9a19e89da5a461d3c6810a3432ae46

The RelatedChange introduced a _make_internal_client() method with an
unused argument 'is_legacy_conf'. This patch completes the original
intention i.e. for the selection of internal client config path to
also be moved to the new method and use the 'is_legacy_conf' arg.
Change-Id: I5075cb446a15edc7f47e83f6aa038c626bd1dd82
RelatedChange: Ia6e1e6a8b58a8476fa16a3c7d45e620c6d7f88e4

The account quota middleware was missing tests for the new quota count
API and it was also only testing the legacy API for quota bytes.
This adds the new APIs quota bytes and quota count to the functional
tests.
Change-Id: I6ebb19c90dfb1cfbe0535ed3860f2319e5153c05

Note that you can still have a "//" in the path with rest_with_last, though.
Change-Id: I171afcd67b162634189b752ff92a4f43484bc12a

Related-Change: I526bbcc59c9eb5923c3784d5d06bc38998cb48db
Change-Id: I820339934f895557ebcde4bb32168b0cfc494948

Account quota metadata such as quota_bytes and quota_count are stored in
the `meta` namespace which users have access to. However, this should be
only available to reseller admins.
This patch adds support for writing the quota metadata to `sysmeta`
namespace, so that it is not accessible by users. The account policy
quota is already using `sysmeta` and has the namespace
`X-Account-Quota-*`, so we are following this pattern.
If present, `X-Account-Quota-Bytes` is always preferred. However, in
order to maintain backwards compatibility, `X-Account-Meta-Quota-Bytes`
will still be honoured if it exists and `X-Account-Quota-Bytes` is not
present.
This also adds some new "legacy" tests to validate backwards
compatibility.
Co-authored-by: Azmain Adib <adib1905@gmail.com>
Co-authored-by: Daanish Khan <daanish1337@gmail.com>
Co-authored-by: Mohammed Al-Jawaheri <mjawaheri02@gmail.com>
Co-authored-by: Nada El-Mestkawy <nadamaged05@gmail.com>
Co-authored-by: Tra Bui <trabui.0517@gmail.com>
Co-authored-by: Chris Smart <distroguy@gmail.com>
Change-Id: Icf7b26023ab5b84136ceaa103fa2797534320f1a

Previously, this would reduce the account's quota to zero, which seems
like the opposite of what the operator intended.
Now, remove the quota, similar to sending an empty quota header.
Change-Id: Ic28752d835e0b970f2baa4e68cbfcde4f500b3d4

Change-Id: Ied0ff6bea7e054fad3fe9579c85d9ae5c9c0b255

- SHA256 mismatches should trip XAmzContentSHA256Mismatch errors,
 not BadDigest. This should include ClientComputedContentSHA256 and
 S3ComputedContentSHA256 elements.
- BadDigest responses should include ExpectedDigest elements.
- Fix a typo in InvalidDigest error message.
- Requests with a v4 authorization header require a sha256 header,
 rejecting with InvalidRequest on failure (and pretty darn early!).
- Requests with a v4 authorization header perform a
 looks-like-a-valid-sha256 check, rejecting with InvalidArgument
 on failure.
- Invalid SHA256 should take precedence over invalid MD5.
- v2-signed requests can still raise XAmzContentSHA256Mismatch errors
 (though they *don't* do the looks-like-a-valid-sha256 check).
- If provided, SHA256 should be used in calculating canonical request
 for v4 pre-signed URLs.
Change-Id: I06c2a16126886bab8807d704294b9809844be086

Found a new way filesystems can break in prod:
 object-auditor: ERROR Trying to audit .../7a7c4af06d2616f23eda274e2ad9c948:
 Traceback (most recent call last):
 File ".../swift/obj/diskfile.py", line 2630, in open
 files = os.listdir(self._datadir)
 OSError: [Errno 117] Structure needs cleaning: .../7a7c4af06d2616f23eda274e2ad9c948
 During handling of the above exception, another exception occurred:
 Traceback (most recent call last):
 File ".../swift/obj/auditor.py", line 238, in failsafe_object_audit
 self.object_audit(location)
 File ".../swift/obj/auditor.py", line 261, in object_audit
 with df.open(modernize=True):
 File ".../swift/obj/diskfile.py", line 2657, in open
 "Error listing directory %s: %s" % (self._datadir, err))
 swift.common.exceptions.DiskFileError:
 Error listing directory .../7a7c4af06d2616f23eda274e2ad9c948:
 [Errno 117] Structure needs cleaning: '.../7a7c4af06d2616f23eda274e2ad9c948'
Change-Id: If6731a6b6b16fbc4eebb61254ed10b53e1767a0f

My home server has more strict controls on /tmp as it run selinux etc.
When running unittests and the default log_filename_prefix deep under
/tmp gets permission denied. It would be better to override this setting
in the tests with a good known tmp location
Change-Id: I6c95ca3a0045a8f268802c6abb633bdfb0e56b73

Change-Id: If5793d1b18cf8f35b8f98206f029f6cc6ca0cf1e

Previously, when handling an object POST, if the proxy got a 202 and
two connection timeouts from the 3 primary backend object servers, it
then went to two handoffs which return 404. The proxy would consider
this to be a quorum of 404s and return the client a 404 response. This
is alarming for the client.
With this patch, the proxy will only treat a 404 response from a
handoff as authoritative if it has an x-backend-timestamp header
(i.e. there's a tombstone on the handoff). POST responses never have
an x-backend-timestamp header so in the scenario described above the
proxy will return a 503.
The Related-Change previously made a similar fix such that 404s from
handoffs are already non-authoritative for object GETs and HEADs
unless they have an x-backend-timestamp header.
Related-Change: Ia832e9bab13167948f01bc50aa8a61974ce189fb
Cloeses-Bug: #2077743
Change-Id: I96f28ab0b2b5f9374c399e8905ee240e7b093f8b

Add unit tests for the proxy object controller to cover object DELETE
and POST scenarios.
Change-Id: I625a4cf03ee9d4a270d60fa2dc9795b36bb36bf1

Some unit tests for the proxy object GET and HEAD path were setup with
insufficient mock responses. The mock connection would raise
StopIteration exceptions once the mock responses were exhausted, which
the object controller would handle as if they were error responses,
potentially distorting the test scenario.
This patch makes the set_http_connect context manager check for
unexpected requests and raise an AssertionError if any are found.
Change-Id: I47774396d9d0a78ebceea6c628c9412b3ad67a11

Running unittests on my home server, even though is a beast, would be
painfully slow. Running them on my laptop ran much faster. I tried
everything and couldn't figure it out.
Chris, co-author, dug in and after we ran out of options he fired up his
packet capture and realised most tests were issuing DNS queries for
`host.`. On my home server, it runs ipv6 dual stack, so these calls
would need to timeout before a test can continue.
On finding this `host.` Ben, another co-author, dug into the code and
found the only reference we have to `host` is in FakeStatsdClient.
Sure enough as I dug a little further it turns out our FakeStatsdClient
used to override a StatsdClient function `_determine_sock_family(self,
host, port)` to stop actually creating a real socket.
However, at some point the StatsdClient was refactored and that method
was renamed and changed. Which leads to a DNS lookup every time we
create a debug_logger as it brings up the socket.
As you can imagine, this happens alot!
This patch overrides the new function that handles to socket creation
`_set_sock_family_and_target(self, host, port)`. Which eliminates the
DNS call all together.
Co-Authored-By: Ben Formosa <bformosa@nvidia.com>
Co-Authored-By: Chris Smart <csmart@nvidia.com>
Change-Id: Ie393075f79447627714692e3f01bb53e967a71e8

Change-Id: Ie73988edf6be0e38d9004bee04ff46c906a759ff

Some tests in test_account_quotas.py registered a response with
FakeSwift during setUp and then subsequently mutated the response's
headers dict. Worse, some tests then mutate the headers again and
re-use the same response.
This relies on the FakeSwift implementation not copying the original
headers when the response is registered, which may not remain the case
(as the author discovered with the Related-Change). It's also an
unnecessary shortcut.
Change-Id: I3217c17936d0c11b03de4a4a172bb2fb0a2be734
Related-Change: I84604a7ea049850ad4d0f0cea2096ab8a98dfb4a

Drive-by: simplify patching of test expirer instance.
Drive-by: fix comment spelling.
Related-Change: Ib151c498ba325f39570e963e5b7948080ffcd3d6
Change-Id: Ia0041c4eedc33a2675890d52cbbc33b9190f1665

Four new metrics have been added into task iteration process:
 'tasks.parse_errors': count of errors when parsing the task object
 'tasks.skipped': count of task objects skipped because it doesn't
 belong to this expirer.
 'tasks.delayed': count of objects is still within the delay
 'tasks.assigned': count of assigned objects to this expirer
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: Ib151c498ba325f39570e963e5b7948080ffcd3d6

Newer versions of python expect to be able to say `with self.lock:`
down in logging.
See https://github.com/python/cpython/commit/74723e11
Change-Id: I30305566d12a1b8be4c8bde4b416b798322a1385

Prior to the Related-Change, non_negative_int would raise a ValueError
if it was passed a positive float. It now casts the float to an int,
which is consistent with the docstring.
This patch adds test assertions to verify this behavior, and similar
behavior of config_positive_int_value.
Change-Id: I5d62e14c228544af634190f16321ee97a8c4211c
Related-Change: I06508279d59fa57296dd85548f271a7812aeb45f

Add unit tests to verify the precedence of access_log_ and log_
prefixes to options.
Add pointers from proxy_logging sections in other sample config files
to the proxy-server.conf-sample file.
Change-Id: Id18176d3790fd187e304f0e33e3f74a94dc5305c

It is currently possible to configure "X-Container-Meta-Quota-Bytes"
and "X-Container-Meta-Quota-Count" on containers, as well as
"X-Account-Meta-Quota-Bytes" on accounts. However, there is no way
to configure an account quota limit for the number of files or
"quota-count". This limitation could potentially allow a user to
exhaust filesystem inodes.
This change introduces the "X-Account-Meta-Quota-Count" header,
allowing users with the ResellerAdmin role to add a quota-count
limit on accounts. The middleware will enforce this limit similarly
to the existing quota-byte limit.
Co-authored-by: Azmain Adib <adib1905@gmail.com>
Co-authored-by: Daanish Khan <daanish1337@gmail.com>
Co-authored-by: Mohamed Hassaneen <mohammedashoor89@gmail.com>
Co-authored-by: Nada El-Mestkawy <nadamaged05@gmail.com>
Co-authored-by: Tra Bui <trabui.0517@gmail.com>
Change-Id: I1d11b2974de8649a111f2462a8fef51039e35d7f

When the proxy passes the container-update headers to the object server
now include the db_state, which it already had in hand. This will be
written to async_pending and allow the object-updater to know more about
a container rather then just relying on container_path attribute.
This patch also cleans up the PUT, POST and DELETE _get_update_target
paths refactoring the call into _backend_requests, only used by these
methods, so it only happens once.
Change-Id: Ie665e5c656c7fb27b45ee7427fe4b07ad466e3e2

The ssl.wrap_socket method has been removed in 3.12.
SSLContext.wrap_socket should now be used.
Change-Id: I6119e054289eac263ff5448d7d118209f98678d9

The proxy may use a sub-request to fetch a container listing response,
for example for updating shard ranges. If it then fails to parse the
response, it should use the sub-request path when logging a warning or
error. Before, the proxy would use the original request path which may
have been to an object.
Change-Id: Id904f4cc0f911f9e6e53d4b3ad7f98aacd2b335d