openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Note that there's a bit of a privilege escalation as prefix-based
tempurls can now be used to perform listings -- but only on containers
with staticweb enabled. Since having staticweb enabled was previously
pretty useless unless the container was both public and
publicly-listable, I think it's probably fine.
This also allows tempurls to be used at the container level, but only
for staticweb responses.
Change-Id: I7949185fdd3b64b882df01d54a8bc158ce2d7032

Change-Id: Ib3fa895ea13a6703b0f146bc8833c4e635976fdd

Some tooling out there, like Ansible, will always call to see if
object-lock is enabled on a bucket/container. This fails as Swift doesn't
understand the object-lock or the get object lock api[0].
When you use the get-object-lock-configuration to a bucket in s3 that
doesn't have it applied it returns a specific 404:
 GET /?object-lock HTTP/1.1" 404 None
 ...
 <?xml version="1.0" encoding="UTF-8"?>
 <Error>
 <Code>ObjectLockConfigurationNotFoundError</Code>
 <Message>Object Lock configuration does not exist for this bucket</Message>
 <BucketName>bucket_name</BucketName>
 <RequestId>83VQBYP0SENV3VP4</RequestId>
 </Error>'
This patch doesn't add support for get_object lock, instead it always
returns a similar 404 as supplied by s3, so clients know it's not
enabled.
Also add a object-lock PUT 501 response.
[0] https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectLockConfiguration.html
Change-Id: Icff8cf57474dfad975a4f45bf2d500c2682c1129

Otherwise, it'd be boto3.
Change-Id: I2961740fd4f3e914675083331f2465591d63b755

We already require swiftclient>=3.2.0, and have for years.
We can stop checking whether it's 1.x
Related-Change: I9842c9975821bda5c7d8bf2fc214480c0c0a5e96
Change-Id: I798904ab66ca10e21b4999ed7f2be74d1b63584c

When using the copy-part API it is expected for s3api to write down an
empty value for X-Object-Sysmeta-S3Api-Etag on segments. This was
ostensibly to prevent writing down an unrelated S3Api-Etag when copying
a part from another MPU the copy transfers object sysmeta. We should
assume a S3Api-Etag w/o X-Static-Large-Object is non-sense, and SLO
should forever expect empty values for it's sysmeta.
Drive-By: consolidate handling of boto2 sigv4 skips
Related-Bug: #2035158
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: Ic6f04a5a6af8a3e65b226cff2ed6c9fce8ce1fa2

When a client sends a malformed http request our server returns a valid http error response with Connection: close and closes the connection.
We want to include a transaction-id and ensure we log details about about the "bad request syntax"
Change-Id: Ic0ee1e4fd4d434d442fcffa68da77e862b37d4c6

Related-Change: I4ecfae2bca6ffa08ad15e584579ebce707f4628d
Related-Change: I1e244c231753b8f4b6f1cf95cb0ae4c3c959ae0f
Change-Id: Ia386736b9b283858931794690538871b6e1ad9c8

Change-Id: I61f141a71eddcac600058d66ddf802306df455c1

Change-Id: Ia564f2ee70f5d04acab1c38e17d1936642a01447

Normally, the proxy object controller would be adding these, but when
encrypted, there won't be any headers in the x-object-meta-* namespace.
Closes-Bug: #1868045
Change-Id: I8e708a60ee63f679056300fc9d68227e46d605e8

Change-Id: I36e42f459a74ed71a1cc57570a564e5562abbae3

unittest.SkipTest suffices.
Change-Id: I11eb73f7dc4a8598fae85d1efca721f69067fb4f

Previously, if a cluster's combined configured max_meta_name_length and
max_meta_value_length constraints were larger than the configured
max_meta_overall_size, we would accidentally go over the overall size
while intending to just test being exactly at the value length-limit.
Change-Id: I42a5287011509e5b43959aab060f9ec7405ae5b9

Depending on test order (and possibly whether there were earlier
failures?) the new tests may trip KeyErrors when trying to get
s3_access_key values. Solution seems to be defining
setUpModule() / tearDownModule() like other functional tests.
Also fix up some Content-MD5 handling; if we're using pre-signed URLs,
we can't provide a Content-MD5.
Change-Id: Ifce72ec255b1b618b9914ce5785d04ee0ebd3b8c
Related-Change: I84494123cfc85e234098c554ecd3e77981f8a096

Previously, clients could use XML external entities (XXEs) to read
arbitrary files from proxy-servers and inject the content into the
request. Since many S3 APIs reflect request content back to the user,
this could be used to extract any secrets that the swift user could
read, such as tempauth credentials, keymaster secrets, etc.
Now, disable entity resolution -- any unknown entities will be replaced
with an empty string. Without resolving the entities, the request is
still processed.
[CVE-2022-47950]
Closes-Bug: #1998625
Co-Authored-By: Romain de Joux <romain.de-joux@ovhcloud.com>
Change-Id: I84494123cfc85e234098c554ecd3e77981f8a096

We've seen S3 clients expecting to be able to send request lines like
 GET https://cluster.domain/bucket/key HTTP/1.1
instead of the expected
 GET /bucket/key HTTP/1.1
Testing against other, independent servers with something like
 ( echo -n $'GET https://www.google.com/ HTTP/1.1\r\nHost: www.google.com\r\nConnection: close\r\n\r\n' ; sleep 1 ) | openssl s_client -connect www.google.com:443
suggests that it may be reasonable to accept them; the RFC even goes so
far as to say
> To allow for transition to the absolute-form for all requests in some
> future version of HTTP, a server MUST accept the absolute-form in
> requests, even though HTTP/1.1 clients will only send them in
> requests to proxies.
(See https://datatracker.ietf.org/doc/html/rfc7230#section-5.3.2)
Fix it at the protocol level, so everywhere else we can mostly continue
to assume that PATH_INFO starts with a / like we always have.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I04012e523f01e910f41d5a41cdd86d3d2a1b9c59

Change-Id: I35cade2c46eb6acb66c064cde75d78173f46864c

We need to support the aforementioned headers in our s3 apis
and raise an InvalidArgumentError if a s3 client makes a request
Change-Id: I2c5b18e52da7f33b31ba386cdbd042f90b69ef97

Previously, this would trip TypeErrors on py310:
 TypeError: 'S3Timestamp' object cannot be interpreted as an integer
Change-Id: I124c1957264c80d28a6b3e852d042cbc8468939c

Sha1 has known to be deprecated for a while so allow the formpost
middleware to use SHA256 and SHA512. Follow the tempurl model and
accept signatures of the form:
 <hex-encoded signature>
or
 sha1:<base64-encoded signature>
 sha256:<base64-encoded signature>
 sha512:<base64-encoded signature>
where the base64-encoding can be either standard or URL-safe, and the
trailing '=' chars may be stripped off.
As part of this, pull the signature-parsing out to a new function, and
add detection for hex-encoded sha512 signatures to tempurl.
Change-Id: Iaba3725551bd47d75067a634a7571485b9afa2de
Related-Change: Ia9dd1a91cc3c9c946f5f029cdefc9e66bcf01046
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Closes-Bug: #1794601 

s3api bucket listing elements currently have LastModified values with
millisecond precision. This is inconsistent with the value of the
Last-Modified header returned with an object GET or HEAD response
which has second precision. This patch reduces the precision to
seconds in bucket listings and upload part listings. This is also
consistent with observation of an aws listing response.
The last modified values in the swift native listing *up* to
the nearest second to be consistent with the seconds-precision
Last-Modified time header that is returned with an object GET or HEAD.
However, we continue to include millisecond digits set to 0 in the
last-modified string, e.g.: '2014年06月10日T22:47:32.000Z'.
Also, fix the last modified time returned in an object copy response
to be consistent with the last modified time of the object that was
created. Previously it was rounded down, but it should be rounded up.
Change-Id: I8c98791a920eeedfc79e8a9d83e5032c07ae86d3

We've known this would eventually be necessary for a while [1], and
way back in 2017 we started seeing SHA-1 collisions [2].
[1] https://www.schneier.com/blog/archives/2012/10/when_will_we_se.html
[2] https://security.googleblog.com/2017/02/announcing-first-sha1-collision.html
UpgradeImpact:
==============
"sha1" has been removed from the default set of `allowed_digests` in the
tempurl middleware config. If your cluster still has clients requiring
the use of SHA-1,
- explicitly configure `allowed_digests` to include "sha1" and
- encourage your clients to move to more-secure algorithms.
Depends-On: https://review.opendev.org/c/openstack/tempest/+/832771
Change-Id: I6e6fa76671c860191a2ce921cb6caddc859b1066
Related-Change: Ia9dd1a91cc3c9c946f5f029cdefc9e66bcf01046
Closes-Bug: #1733634 

Closes-Bug: 1966396
Change-Id: I253d8e3e8678fad3fde43259ed3225df4048a458

Change-Id: Idcda76f7a880a18c3bac699e0fb2435e4a54abbd

Co-Authored-By: Florent Vennetier <florent.vennetier@ovhcloud.com>
Change-Id: I635bc91faa7709f9df9cdf3aec157a21c08923ca

Specifically, parameters that may contain non-ASCII characters,
such as the prefix and marker to list current uploads.
Change-Id: Icfae68825f94ddf2412c0274c3d500e265117e8e

Previous problems included:
 - returning wsgi strings quoted assuming UTF-8 on py3 when initiating
 or completing multipart uploads
 - trying to str() some unicode on py2 when listing parts, leading to
 UnicodeEncodeErrors
Change-Id: Ibc1d42c8deffe41c557350a574ae80751e9bd565

Trying to copy an object with non-ASCII characters in its name results
in, depending on the pipeline:
- an error code 412 because of a badly urlencoded path
- an error code 500 "TypeError: Expected a WSGI string"
This commit fixes the problem by calling str_to_wsgi on the object name
after it has been urldecoded. We do not need to call this on the
container name because it is supposed to contain only ASCII characters.
Change-Id: If837d4e55735b10a783c85d91f37fbea5e3baf1d

Not all v1 auth systems use an acct:user format; s3api tests should not require it.
Be a little more tolerant of listing consistency issues when resetting.
Tolerate s3api /info results returning strings instead of ints.
Related-Change: I4a46bd650a53f88c642d402e697869df28bd2fd3
Change-Id: I8f2f247dd113ad637b17d241133b14c35cadecae

We started tolerating deleted containers still showing up in account
listings recently, but we've still seen the occassional gate failure.
Change-Id: I2f0c82981e4323e5e210119f917b4645670b44d3
Related-Change: I9217ba49d65cc29496b1532644bbef6b736074b8

Change-Id: Id1ee51fea6b754042ebe5204517dd91865ffbc28

Change-Id: I9217ba49d65cc29496b1532644bbef6b736074b8

Change-Id: I8ce73e2e21e9216484130ba3bd1e77b45eb1d77c

Change-Id: I19ca860deaa6dbf388bdcd1f0b0f77f72ff19689

Some middlewares (notably staticweb) use the absence of a REMOTE_USER to
determine that a request was unauthenticated and as a result should be
handled differently. This could cause problems for S3 requests that
were authenticated via s3api's custom auth logic, including
* server errors when a container listing request gets handled by
 staticweb or
* losing storage policy information because staticweb copied the request
 environment.
Change-Id: Idf29c6866fec7b413c4369dce13c4788666c0934
Closes-Bug: #1833287
Related-Change: I5fe5ab31d6b2d9f7b6ecb3bfa246433a78e54808

md5 is not an approved algorithm in FIPS mode, and trying to
instantiate a hashlib.md5() will fail when the system is running in
FIPS mode.
md5 is allowed when in a non-security context. There is a plan to
add a keyword parameter (usedforsecurity) to hashlib.md5() to annotate
whether or not the instance is being used in a security context.
In the case where it is not, the instantiation of md5 will be allowed.
See https://bugs.python.org/issue9216 for more details.
Some downstream python versions already support this parameter. To
support these versions, a new encapsulation of md5() is added to
swift/common/utils.py. This encapsulation is identical to the one being
added to oslo.utils, but is recreated here to avoid adding a dependency.
This patch is to replace the instances of hashlib.md5() with this new
encapsulation, adding an annotation indicating whether the usage is
a security context or not.
While this patch seems large, it is really just the same change over and
again. Reviewers need to pay particular attention as to whether the
keyword parameter (usedforsecurity) is set correctly. Right now, all
of them appear to be not used in a security context.
Now that all the instances have been converted, we can update the bandit
run to look for these instances and ensure that new invocations do not
creep in.
With this latest patch, the functional and unit tests all pass
on a FIPS enabled system.
Co-Authored-By: Pete Zaitcev
Change-Id: Ibb4917da4c083e1e094156d748708b87387f2d87

No version of eventlet that I'm aware of hasany sort of support for
eventlet.wsgi.WRITE_TIMEOUT; I don't know why we've been setting that.
On the other hand, the socket_timeout argument for eventlet.wsgi.Server
has been supported for a while -- since 0.14 in 2013.
Drive-by: Fix up handling of sub-second client_timeouts.
Change-Id: I1dca3c3a51a83c9d5212ee5a0ad2ba1343c68cf9
Related-Change: I1d4d028ac5e864084a9b7537b140229cb235c7a3
Related-Change: I433c97df99193ec31c863038b9b6fd20bb3705b8