openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Change-Id: Ibd111773a496e8e3b1fc1577f40aa69f4328139b

Change-Id: If15c3fee706d295de678f812839a5d0759089ccf

This has been available since py32 and was backported to py27; there
is no point in us continuing to carry the old idiom forward.
Change-Id: I21f64b8b2970e2dd5f56836f7f513e7895a5dc88

Last time we did this was nearly 4 years ago; drag ourselves into
something approaching the present. Address a few new pyflakes issues
that seem reasonable to enforce:
 E275 missing whitespace after keyword
 E231 missing whitespace after ','
 E721 do not compare types, for exact checks use `is` / `is not`,
 for instance checks use `isinstance()`
Main motivator is that the old hacking kept us on an old version
of flake8 et al., which no longer work with newer Pythons.
Change-Id: I54b46349fabb9776dcadc6def1cfb961c123aaa0

While we use requests-mock, we don't need the pytest plugin.
Change-Id: If14e4d2c1af2efcbc99e9b6fe10973a7eb94d589

Using an antiquated distro to keep us on an EOL'ed version of python
that *just so happens* to have had support dropped for it around the
same time virtualenv removed the ability to create py27 environments
doesn't seem like the right way to address the problem.
Make sure we also include the tox<4 pin that we're using project-wide
so we don't get some nonsense about "Multiple top-level packages
discovered in a flat-layout".
Related-Change: https://review.opendev.org/c/openstack/swift/+/881035
Change-Id: I32a161e555179ca34d306ac37e4097611853e36b

Keep them when running via tox, though.
Related-Change: https://review.opendev.org/c/openstack/swift/+/851099
Change-Id: I1ddd1dc8a3b35a588a8ecc5667dae4da33340712

We need to install swift to build docs anyway, so make sure we include
swift's requirements as well as the docs-building requirements.
Otherwise, swift's requirements will not be subject to the specified
constraints file.
Change-Id: If33532232572e802b2099f39f4231ee6add6d5a9

For tox 3.x and earlier, passenv was a space-separated list; as of tox
4.0.0, it's comma-separated. For a while, our spaces would be silently
included in the now-one-and-only passenv value parsed (which wasn't
great, but mostly just caused confusion) -- as of tox 4.0.6, however, it
became a hard error, and all tests would fail like
 pass_env values cannot contain whitespace, use comma to have multiple
 values in a single line, invalid values found 'SWIFT_* *_proxy'
Unfortunately, we don't really know what versions of tox all our various
stakeholders might want/need to use (though we previously set a
minversion of 2.3.2). We might be able to spread values over multiple
lines to make it compatible with both tox 3 *and* tox 4, but I'm fairly
certain *_proxy was only included for some variables that are recent
versions of tox include by default anyway, so just increase our
minversion (which was too low, anyway -- allowlist_externals which we
already configure was added in 3.18.0) and get rid of *_proxy.
FWIW, python-swiftclient was already specifying 3.18.0 as a minversion,
so I expect the new minversion to not be a problem.
Also, add ./.functests to a bunch of allowlist_externals, as newer tox
is more strict about that sort of thing.
Drop skipsdist in a bunch of places so we can import swift from func
tests and docs. (Still not sure why I don't see us hitting a similar
problem for unit tests...)
Change-Id: I4be1e86e3291ad1619c695fb93d7cadf053b556d

nose has not seen active development for many years now. With py310, we
can no longer use it due to import errors.
Also update lower contraints
Closes-Bug: #1993531
Change-Id: I215ba0d4654c9c637c3b97953d8659ac80892db8

Change-Id: I70dec389c1ac608a9d2767d162bd5edc315e564b

Now that OpenStack writ large has dropped support for py36,
we're going to have to manage our constraints ourselves
more, just like we did for py27.
Change-Id: I0973be6692e3a9b871eb61cbf759e13644a25107

Change-Id: I58d4b5a00e97785584c6d3bd8b06243f481c1934

Change-Id: I8db90ce8198aad720a644186d6a122df6f0058ae

Change-Id: I495fb1ec2394130c7274368662b58212ca375854

Tox trying to install latest versions for building docs which may
not be supported by stable and lower branches, so should be
restricted by respective version's upper-constraints.txt
Change-Id: I599d4ea430cc1e65bb50d0481617b005febf6de2

Add the librsvg2-* package to bindep.txt. This package has been
required to build docs since the Related-Change. The package is added
to a new 'doc' profile in bindep.txt; if missing, the package will be
reported by:
 bindep doc
The 'doc' profile is added to the tox 'bindep' env command so that, if
missing, the package will always be reported by:
 tox -e bindep
Change-Id: I6c60c31ca8002133ab77f05f59359b25315b299c
Related-Change: I26cefda80d3234df68d7152b404e0a71da74ab90
Closes-Bug: 1903038

UPPER_CONSTRAINTS_FILE is old name and deprecated
-https://zuul-ci.org/docs/zuul-jobs/python-roles.html#rolevar-tox.tox_constraints_file
This allows to use lower-constraints file as more
readable way instead of UPPER_CONSTRAINTS_FILE=<lower-constraints file>.
[1] https://review.opendev.org/#/c/722814/
Change-Id: I7e3cdef5b1bcb28019055d29dbd95073e92882fc

This patch removed the separate s3api, staticweb functional tests
gate jobs and added them across all other functional test jobs.
Change-Id: Ie1c606132a054defc2b3cc14a66031090e7b8449

New flake8 came out with new & improved rules. Ignore E741; it would be
too much churn. Fix the rest.
Change-Id: I9125c8c53423232309a75cbcc5b695b378864c1b

The global upper constraints repo will no longer track py2 packages,
meaning upper constraints for the py2.7 runtime will need to be handled
locally.
This adds an in-repo file for upper constraints enforcement using the
last py27 constraints from openstack/requirements to validate
functionality.
Change-Id: I624f914c6155178cbe474f1eae2871e2aef721eb
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>

Change-Id: Ia968ec7375ab346a2155769a46e74ce694a57fc2

The repo is Python using both Python 2 and 3 now, so update hacking to
version 2.0 which supports Python 2 and 3. Note that latest hacking
release 3.0 only supports version 3.
Fix problems found.
Remove hacking and friends from lower-constraints, they are not needed
for installation.
Change-Id: I9bd913ee1b32ba1566c420973723296766d1812f

Swift servers can now be seamlessly reloaded by sending them a SIGUSR1
(instead of a SIGHUP). The server forks off a synchronized child to
wait to close the old listen socket(s) until the new server has started
up and bound its listen socket(s). The new server is exec'ed from the
old one so its PID doesn't change. This makes Systemd happier, so a
ReloadExec= stanza can now be used.
The seamless part means that incoming connections will alwyas get
accepted either by the old server or the new one. This eliminates
client-perceived "downtime" during server reloads, while allowing the
server to fully reload, re-reading configuration, becoming a fresh
Python interpreter instance, etc. The SO_REUSEPORT socket option has
already been getting used, so nothing had to change there.
This patch also includes a non-invasive fix for a current eventlet bug;
see https://github.com/eventlet/eventlet/pull/590
That bug prevents a SIGHUP "reload" from properly servicing existing
requests before old worker processes close sockets and exit. The
existing probtests missed this, but the new ones, in this patch, caught
it.
New probe tests cover both old SIGHUP "reload" behavior as well as the
new SIGUSR1 seamless reload behavior.
Change-Id: I3e5229d2fb04be67e53533ff65b0870038accbb7

...because otherwise, I think they might be running under py3 right now.
Change-Id: I96ad9e5ed87a7e08d06e889233730cec476d3af7

Change-Id: I5c856012a04ef8f910c28b0248ea7bcda47e4816

This patch adds a `pdf-docs` tox target that will build
PDF versions of our docs. As per the Train community goal:
 https://governance.openstack.org/tc/goals/selected/train/pdf-doc-generation.html
Add sphinxcontrib-svg2pdfconverter to doc/requirements.txt
to convert our SVGs.
Story: 2006122
Task: 35515
Change-Id: I26cefda80d3234df68d7152b404e0a71da74ab90

We were (indirectly) importing swiftclient (and therefore requests and
urllib3) before doing our eventlet monkey-patching. This would lead
boto3 (which digs an SSLContext out of urllib3) to trip RecursionErrors
on py3 similar to
 >>> from ssl import SSLContext, PROTOCOL_SSLv23
 >>> import eventlet
 >>> eventlet.monkey_patch(socket=True)
 >>> SSLContext(PROTOCOL_SSLv23).options |= 0
 Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/usr/lib/python3.6/ssl.py", line 465, in options
 super(SSLContext, SSLContext).options.__set__(self, value)
 File "/usr/lib/python3.6/ssl.py", line 465, in options
 super(SSLContext, SSLContext).options.__set__(self, value)
 File "/usr/lib/python3.6/ssl.py", line 465, in options
 super(SSLContext, SSLContext).options.__set__(self, value)
 [Previous line repeated 330 more times]
 RecursionError: maximum recursion depth exceeded while calling a Python object
Change-Id: I4bb59edd87336597791416c4f2a096efe0e72fe3

Change-Id: I9a754f1eb06debbe16800a05fc8e969af2f89043

Change-Id: I8c2517599a6d1cdf68050dc7acb039f09875fd53

test_bucket.py is proving somewhat problematic.
Change-Id: I5b337ef66a23fc989762801dd6a5ba1ed903f57b

Drive-by: fix an issue where tf.config wasn't initialized when we decide
whether to skip test_dlo_referer_on_segment_container.
Change-Id: Iad8b3ab40dccb23847cde5c09f55627da0963fe0

Change-Id: Ia9ae0fc226dfc9b40157faebac100c10a9180c62

Yes, it really is just the import.
Change-Id: Id7e2891b6600af102176eaa2e87f81b528adde8d

The api documentation is now published on docs.openstack.org instead
of developer.openstack.org. Update all links that are changed to the
new location.
Note that the Swift API lives at /object-store and not /object-storage.
Note that redirects will be set up as well but let's point now to the
new location.
For details, see:
http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007828.html
Change-Id: Ie38357e4c278335c35d186708573bb6bdabaa012

Drive-by: remove some py2/py3 code branches where encoding/decoding
should always succeed.
Change-Id: Iabfa157f2b20e6fd650a441e9c0f6163478ce147

Change-Id: I2929ce1f8e08ae5b4059605d09189c8521852423

Drive-by: Tighten domain_remap assertions on listings, which required
that we fix proxy pipeline placement. Add a note about it to the sample
config.
Change-Id: I41835148051294088a2c0fb4ed4e7a7b61273e5f

For more detail, see http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006478.html
Change-Id: I95114c4aa670c07491d5a15db2341f65cb0d1344

This patch ports the tempurl middleware over to PY3. We use
an "all-native" string model, where we convert WSGI strings
as soon as we see them. It helps to deal with HMAC.
Aaand, we fix formpost along the way. It _clearly_ was
doing a wrong thing, encoding the same body several times
if we had several keys. On py2 it wasn't noticeable, but
on py3 this breaks, because the bytes type cannot be encoded
again.
Change-Id: I69974cc8a39731c980b54137b799a36b2e63a44a