openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Previously some tests in /test/functional/test_staticweb.py assumed
that domain_remap middleware was configured with the default
storage_domain 'example.com'. As a result those tests would fail if
run against a cluster with domain_remap middleware installed but not
having an 'example.com' storage_domain.
This patch allows a storage_domain to be configured in the test.conf
file so that the tests may be successfully run against any cluster. If
the option is not set then the tests will be skipped.
To ensure that the in process test job func-domain-remap-staticweb
still runs all the tests in test_staticweb.py, its config loader is
modified to add the required storage_domain option to the test config
with the same value used by the domain_remap middleware.
Related-Change: I6bc70039c1b99b8a5bf5a30e9c6877d4f0aa638d
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: Ifab84fc3ef66b9790e96ebb8e5c8061a12636045

This attempts to import openstack/swift3 package into swift upstream
repository, namespace. This is almost simple porting except following items.
1. Rename swift3 namespace to swift.common.middleware.s3api
1.1 Rename also some conflicted class names (e.g. Request/Response)
2. Port unittests to test/unit/s3api dir to be able to run on the gate.
3. Port functests to test/functional/s3api and setup in-process testing
4. Port docs to doc dir, then address the namespace change.
5. Use get_logger() instead of global logger instance
6. Avoid global conf instance
Ex. fix various minor issue on those steps (e.g. packages, dependencies,
 deprecated things)
The details and patch references in the work on feature/s3api are listed
at https://trello.com/b/ZloaZ23t/s3api (completed board)
Note that, because this is just a porting, no new feature is developed since
the last swift3 release, and in the future work, Swift upstream may continue
to work on remaining items for further improvements and the best compatibility
of Amazon S3. Please read the new docs for your deployment and keep track to
know what would be changed in the future releases.
Change-Id: Ib803ea89cfee9a53c429606149159dd136c036fd
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>

Change-Id: Iea629d5a08aa3d94e097fcdab28f94511b262fcf

Change-Id: I9da8f6a56ffb06e89b8da8c06b26ebc8452b652d
Closes-Bug: #1530254 

This patch adds test cases for PUT, DELETE, GET, HEAD, POST and OPTIONS
requests to accounts, containers and objects using various combinations
of users/projects, roles and/or service tokens.
Change-Id: Iea8141ac74ad949a3ae7fa47fda3135d0f2612f6

Change-Id: Iec86b488d71553c295afe7098822ce2046df9546

Make the sample test config project for tester5 be test5
to match the configured account in proxy-server.conf-sample.
That way the sample test config will work if uncommented.
Change-Id: I57144fe153775de51c950cfc101fcdf34dfac5e8

This change is in support of Composite Tokens and Service Accounts
(see http://specs.openstack.org/openstack/swift-specs/specs/in_progress/
service_token.html)
During coding, minor changes were made compared to the original
specification. See https://review.openstack.org/138771 for these changes.
DocImpact
Change-Id: I6072b4efb3a479a8e0cc2d9c11ffda5764b55e30

The functional tests devstack gate job currently skips tests
that require keystone v3 API. In conjunction with changes to
devstack [0] this patch will cause the gate job to use keystone
v3 API and stop skipping those tests.
Devstack sets auth version in its /etc/swift/test.conf by
uncommenting the value found in test/sample.conf, so the change
in this patch is sufficient to cause the v3 API to be used.
The devstack patch [0] sets the correct auth url for the v3 API
and creates a fourth test account that is required for the
currently skipped tests to run.
[0] change id I557de01bf196075f2f3adcdf4dd1b43756d8a0ae
Change-Id: I44e159b939f5e3fc3780612823d12262013ea793

Corrects spelling errors found in comments.
Change-Id: I228a888e3f256569ea32ef1613092dbd63e13c62

The keystoneauth middleware supports cross-tenant access
control using the syntax <tenant>:<user> in container ACLs,
where <tenant> and <user> may currently be either a unique
id or a name. As a result of the keystone v3 API introducing
domains, names are no longer globally unique and are only
unique within a domain. The use of unqualified tenant and
user names in this ACL syntax is therefore not 'safe' in a
keystone v3 environment.
This patch modifies keystoneauth to restrict cross-tenant
ACL matching to use only ids for accounts that are not in
the default domain. For backwards compatibility,
names will still be matched in ACLs when both the requesting
user and tenant are known to be in the default domain AND the
account's tenant is also in the default domain (the default
domain being the domain to which existing tenants are
migrated).
Accounts existing prior to this patch are assumed to be for
tenants in the default domain. New accounts created using a
v2 token scoped on the tenant are also assumed to be in the
default domain. New accounts created using a v3 token scoped
on the tenant will learn their domain membership from the
token info. New accounts created using any unscoped token,
(i.e. with a reselleradmin role) will have unknown domain
membership and therefore be assumed to NOT be in the default
domain.
Despite this provision for backwards compatibility, names
must no longer be used when setting new ACLs in any account,
including new accounts in the default domain.
This change obviously impacts users accustomed to specifying
cross-tenant ACLs in terms of names, and further work will be
necessary to restore those use cases. Some ideas are
discussed under the bug report. With that caveat, this patch
removes the reported vulnerability when using
swift/keystoneauth with a keystone v3 API.
Note: to observe the new 'restricted' behaviour you will need
to setup keystone user(s) and tenant(s) in a non-default domain
and set auth_version = v3.0 in the auth_token middleware config
section of proxy-server.conf. You may also benefit from the
keystone v3 enabled swiftclient patch under review here:
https://review.openstack.org/#/c/91788/
DocImpact
blueprint keystone-v3-support
Closes-Bug: #1299146
Change-Id: Ib32df093f7450f704127da77ff06b595f57615cb

The problem arises when running functests against pre-existing
clusters configured with self-signed certificates. As far as
security goes, doing that is rather useless, but it happens.
Such testing worked previously, but with the switch to Requests
in the client, we must plumb "insecure" flag. There's no other
way to do it in Requests, no environment variable, anything.
Change-Id: I94a61b60de8ee0529652c8c4a1ab9894c7cdb589

Use constrainst from the new "swift-constraints" section of test.conf,
fall back to those found in the response to the /info API call,
ultimately falling back to the constraints module's effective
constraints.
Change-Id: Iea01c9c4b5148faa10004a240df411cbe7336a6a

Looks like we are keeping at least four lists of the default
constraints in various places, and when we added the max_header_size
in initial Static Large Object Support commit (5d73da15) we failed to
add it to the two locations in the test tree (test/sample.conf and
test/functional/tests.py).
With this commit we take a small step to consolidate the list of
constraints to the constraints module by removing the list from the
tests module (note we still have them listed in the sample swift.conf
file and the test module's sample.conf file), document the missing
reference in the test module's sample.conf file, and, while we're at
it, use the SWIFT_CONF_FILE reference where possible.
Change-Id: Ic4fa5f1aedec9bc7a7b43ce1a2ca3cdebce6171f

Currently probetests take advantage of a number of assumptions about the SUT.
Unfortunately after some time with a working SAIO, configuration drift may
result in a system that is no longer compatible with these assumptions. To
help weary developers more quickly identify the changes they've made since
they last ran probetests successfully, some handy validators have been added
to test.probe.common
Additionally a new option 'validate_rsync' in test.conf, when enabled, will
run a series of up front validations during the setup of each probetest by
inspecting the ring, the mounted devices, and the rsync exports ("modules") in
order to ensure that when probetests fail the do so early and with specific
complaints.
To preserve existing failures, the option is disabled by default.
Change-Id: I2be11c7e67ccd0bc0589c360c170049b6288c152

While poking at such problems with authentication other than swauth
it was useful to allow stock functional tests running back-to-back
at two clusters, with tempauth and Keystone.
When running the legacy tests, this version packs account into user
the way it was down previously. This way we do not need to repack
them before every call to get_auth. The downside is additional work
to be done when setting ACLs.
Change-Id: Ieb1d9227cb88977ecd2c39825039bc4be8afae0b

Currently the timeout for a wsgi server successfully binding to a port
and for a probetest background service to finish starting are hard coded
to 30 seconds. While a reasonable default for most configurations, a
small virtualized environment may need a little more time in order for
probe tests to complete successfully.
This patch adds a 'bind_timeout' option to the DEFAULT section of the
main wsgi servers' config. Also a new [probe_test] section and
'check_server_timeout' option to test.conf
DocImpact
Change-Id: Ibcaff153c7633bbf32e460fd9dbf04932eddb56f

Change previously hard-coded constants into config variables. This
allows deployers to tune their cluster more specifically based on
their needs. For example, a deployment that uses direct swift access
for public content may need to set a larger header value constraint to
allow for the full object name to be represented in the Content-
Disposition header (which browsers check to determine the name of a
downloaded object).
The new settings are set in the [swift-constraints] section of
/etc/swift/swift.conf. Comments were also added to this config file.
Cleaned up swift/common/constraints.py to pass pep8 1.3.3
Funtional tests now require constraints to be defined in /etc/test.conf or in
/etc/swift/swift.conf (in the case of running the functional tests against a
local swift cluster). To have any hope of tests passing, the defined
constraints must match the constraints on the tested cluster.
Removed a ton of "magic numbers" in both unit and functional tests.
Change-Id: Ie4588e052fd158314ddca6cd8fca9bc793311465

bug 701248
bug 819303
This change makes syslog optional, by providing a unit test
config with the ability to cause test/unit/__init__.py to
replace SysLogHandler with a fake logging class. The
default behavior is unchanged.
FakeLogger now inherits directly from Handler and mocks out
its API - this was a backward-compatibility issue.
Change-Id: I653dec148c4f6cf81759de03964c6a3290c1a290