openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Updates the proxy-server.conf-sample and docs to use
the new Keystoneclient middleware class name.
Change-Id: I3727f7b7328a2513347b8ef257c270126df36d7b

Change-Id: I720a0e0e886d25eec52556e8ce93bf9c49f6b452

Two improvements: first, document that the container-sync process
connects to the remote cluster's proxy server, so outbound
connectivity is required.
Second, rewrite the behind-the-scenes container-sync example and add
some ASCII-art diagrams.
Fixes bug 1068430.
Bonus fix of docstring in wsgi.py to squelch a sphinx warning.
Change-Id: I85bd56c2bd14431e13f7c57a43852777f14014fb

The determination of the client IP looked at the X-Cluster-Client-Ip
and X-Forwarded-For headers in the incoming HTTP request. This is
trivially spoofable by a malicious client, so there's no security
gained by having the check there.
Worse, having the check there provides a false sense of security to
cluster operators. It sounds like it's based on the client IP, so an
attacker would have to do IP spoofing to defeat it. However, it's
really just a shared secret, and there's already a secret key set
up. Basically, it looks like 2-factor auth (IP+key), but it's really
1-factor (key).
Now, the one case where this might provide some security is where the
Swift cluster is behind an external load balancer that strips off the
X-Cluster-Client-Ip and X-Forwarded-For headers and substitutes its
own. I don't think it's worth the tradeoff, hence this commit.
Fixes bug 1068420 for very small values of "fixes".
DocImpact
Change-Id: I2bef64c2e1e4df8a612a5531a35721202deb6964

The auth_token middleware has moved, updating the doc.
Change-Id: I8133b876a53d61bf94169cf08c8b1fa6bbf9681a

There are 3 sections in there, all useless.
Section 1 tells you how to install Swift packages from the swift-core
PPA. However, the latest version there is ancient.
Section 2 tells you how to build your own Swift packages. However, it
talks about getting the source code from the "debian" branch in bzr,
which is obviously really old.
Section 3 tells you how to take the packages from section 2 and
install them. This isn't too out-of-date, but since section 2 doesn't
work any more, section 3 is useless.
Since stale docs are worse than no docs, there's no current
information in this document, and bringing it up-to-date requires a
whole pile of work, I've chosen to delete it entirely.
Also pulled out a couple references to the PPA elsewhere.
Fixes bug 917385.
Fixes bug 1026145.
Change-Id: I510bd8619531fe110419e5488bd20d3602868d66

Change-Id: Iab43ac342faf3722ccab1ed034f08752dbbfaae1

I know it's just TempAuth, but bug #959953 just caught my eye as
something interesting to solve.
This does a best guess on the storage URL to return for a given
request. It allows $HOST to be used in the storage URL configuration,
where $HOST will resolve to scheme://host:port. It bases the scheme
on how the server is running or on storage_url_scheme if set. The
host:port comes from the request's Host header if it exists, and
falls back to the WSGI SERVER_NAME:SERVER_PORT otherwise.
Fixes: bug #959953
DocImpact
Change-Id: Ia494bcb99a04490911ee8d2cb8b12a94e77820c5

Change-Id: I99d9ddfbcad0f88e75c49235c8317ea97237d4e4

Change-Id: I77e1d7fdcf217826402beeb7d583e3c7279c416c

Adds a -p option to swift-dispersion-report that will output the
partitions missing copies to standard error. Another thing we've been
meaning to add for forever. It's useful when you want to do some
further research on whether a partition truly has fewer copies or if
they're just somewhere else in the cluster due to a ring change, for
instance.
DocImpact
Change-Id: I5e47aa5818483ecc34b39ef6f8cd83ad312ed9a0

Change-Id: I78a5c109c9440df752e390698502f57d4392fb67

Change-Id: I894473994cdfea0996ad16e7619aff421f604abc

It's not sufficient to just look at swift.object-updater.successes to
see the async_pending unlink rate. There are two different spots where
unlinks happen: one when an async_pending has been successfully
processed, and another when the updater notices multiple
async_pendings for the same object. Both events are now tracked under
the same name: swift.object-updater.unlinks.
FakeLogger has now sprouted a couple of convenience methods for
testing logged metrics.
Fixed pep8 1.3.3's complaints in the files this diff touches.
Also: bonus speling and, grammar fixes in the admin guide.
Change-Id: I8c1493784adbe24ba2b5512615e87669b3d94505

Fixes bug 1064249
sys.path should be extended not appended to
Change-Id: I3404a50b16b843b47f5082676029a4dd3b03f812

Change-Id: Ie566e712ba0d3dca543e99efbdb576b19623e73c

This patch introduces a new user format to tempauth which starts with
"user64_" and in which the account and user are base64 encoded (sans
padding equal signs):
user64_<account_b64>_<user_b64> = <key> [group] [group] [...] [storage_url]
This allows unambiguous parsing and adds the ability to have a user or
account name containing underscores.
The handling of tempauth configuration is fully backward compatible and
this only adds new, optional functionality.
Also beefed up unit tests to get full coverage of the user parsing code
in tempauth's __init__().
Change-Id: Id06304934ab8055a28921f2e1aa3c58a2036d8f8

This change replaces WebOb with a mostly compatible local library,
swift.common.swob. Subtle changes to WebOb's API over the years have been a
huge headache. Swift doesn't even run on the current version.
There are a few incompatibilities to simplify the implementation/interface:
 * It only implements the header properties we use. More can be easily added.
 * Casts header values to str on assignment.
 * Response classes ("HTTPNotFound") are no longer subclasses, but partials
 on Response, so things like isinstance no longer work on them.
 * Unlike newer webob versions, will never return unicode objects.
Change-Id: I76617a0903ee2286b25a821b3c935c86ff95233f

Change-Id: If45f77fda269ae6e251579542e70eb71bd11fe2a

Change-Id: I8130e258d77f47bfa28ecf79a51a6e34671ada22

This moves search_devs into RingBuilder to make it accessible to other utils
that need to search the builder. Along the same lines this also adds a
load() call to get a RingBuilder instance when working with the builder files.
- This adds python-mock >= 0.7 as a dependency for unittests. On Ubuntu
 10.04 you'll have to pip install it, on 12.04 you can apt-get install
 it. Fedora 17+ should be able to yum install it.
- new pep8 compliance
- Fixed a small issue (undefined var) in swift-ring-builder when remove was
called but failed to find a match.
Change-Id: I2e02684235aa2f4e901a00858ae037091594c545

Removed many StatsD logging calls in proxy-server and added
swift-informant-style catch-all logging in the proxy-logger middleware.
Many errors previously rolled into the "proxy-server.<type>.errors"
counter will now appear broken down by response code and with timing
data at: "proxy-server.<type>.<verb>.<status>.timing". Also, bytes
transferred (sum of in + out) will be at:
"proxy-server.<type>.<verb>.<status>.xfer". The proxy-logging
middleware can get its StatsD config from standard vars in [DEFAULT] or
from access_log_statsd_* config vars in its config section.
Similarly to Swift Informant, request methods ("verbs") are filtered
using the new proxy-logging config var, "log_statsd_valid_http_methods"
which defaults to GET, HEAD, POST, PUT, DELETE, and COPY. Requests with
methods not in this list use "BAD_METHOD" for <verb> in the metric name.
To avoid user error, access_log_statsd_valid_http_methods is also
accepted.
Previously, proxy-server metrics used "Account", "Container", and
"Object" for the <type>, but these are now all lowercase.
Updated the admin guide's StatsD docs to reflect the above changes and
also include the "proxy-server.<type>.handoff_count" and
"proxy-server.<type>.handoff_all_count" metrics.
The proxy server now saves off the original req.method and proxy_logging
will use this if it can (both for request logging and as the "<verb>" in
the statsd timing metric). This fixes bug 1025433.
Removed some stale access_log_* related code in proxy/server.py. Also
removed the BaseApplication/Application distinction as it's no longer
necessary.
Fixed up the sample config files a bit (logging lines, mostly).
Fixed typo in SAIO development guide.
Got proxy_logging.py test coverage to 100%.
Fixed proxy_logging.py for PEP8 v1.3.2.
Enhanced test.unit.FakeLogger to track more calls to enable testing
StatsD metric calls.
Change-Id: I45d94cb76450be96d66fcfab56359bdfdc3a2576

Change-Id: Id8872c581ed23378a8e14cbf3bf049b5c0d21577

To tell when replication for a device has finished, it's important to
know when the replicator is removing objects. This was previously
handled for the object-replicator
(object-replicator.partition.delete.count.<device> and
object-replicator.partition.update.count.<device> metrics) but not the
account and container replicators.
This patch extends the existing DB removal count metrics to make them
per-device. The new metrics are:
 account-replicator.removes.<device>
 container-replicator.removes.<device>
There's also a bonus refactoring and increased test coverage of the DB
replicator code.
Change-Id: I2067317d4a5f8ad2a496834147954bdcdfc541c1

Fix a PEP8 issues in doc/source/conf.py to make the code pretty.
Change-Id: I71d36ebb59d44ba145ca13e4f4827139f0466d78

Added "-p" to mkdir cmds in a "set -e" script (/etc/rc.local).
/var/log/swift gets owned by the group "adm", but isn't group writeable
by default, so the find/rm in "resetswift" fails. So I added a chmod to
add group-write for that directory tree.
Added commands for installing tox.
Change-Id: Ic141491272a83a9233cd6a6a70c3ef74411f3f51

The Admin Guide now contains information about the ring serialization
change (and importantly, how to downgrade, if necessary).
Also added container-server conf var, "allow_versions" to the Deployment
Guide.
Also changed description of proxy-server conf var,
"max_containers_whitelist" to say it contains "account names" not
"account hashes".
Change-Id: Ib23c6118cc5195cc04765afd28e442e4c735f0d4

Change-Id: I00bf2e20c2683d0440ca425461e0ab87197ee720

bug fixes for swift 1033089
Swift 1.6.0 All in One setup uses rc.local to create a set of
directories like the following:
 /var/cache/swift /var/cache/swift2 /var/cache/swift3
 /var/cache/swift4
 /var/run/swift
without -p for mkdir command, when some of these directories
already exist, the rc.local fails so that the next set of
command won't execute. This causes swift fail to start because
of /var/run/swift does not exist. Adding -p to both mkdir
command solves this problem.
Change-Id: Ia26c9a857723747008293c0631c9acc1b58eca7e

We don't want to use pickle as it can execute arbitrary code. JSON is
safer. However, note that it supports serialization for only some
specific subset of object types; this should be enough for what we need,
though.
To avoid issues on upgrades (unability to read pickled values, and cache
poisoning for old servers not understanding JSON), we add a
memcache_serialization_support configuration option, with the following
values:
 0 = older, insecure pickle serialization
 1 = json serialization but pickles can still be read (still insecure)
 2 = json serialization only (secure and the default)
To avoid an instant full cache flush, existing installations should
upgrade with 0, then set to 1 and reload, then after some time (24
hours) set to 2 and reload. Support for 0 and 1 will be removed in
future versions.
Part of bug 1006414.
Change-Id: Id7d6d547b103b4f23ebf5be98b88f09ec6027ce4

A warning log line is emitted whenever the proxy has to use a handoff
node. Monitoring these warnings can indicate a problem within your
cluster; however, you can disable these log lines by setting the
proxy conf's log_handoffs to false.
While working on this, I also noticed why many proxy log lines did
not have txn_id and client_ip -- subcoroutines. Now the logger thread
locals are copied to the subcoroutines.
Change-Id: Ibac086e1b985f566c068d083620287509de35da8

Fixes bug #1024477
Change-Id: Id09bbff6bb4a5d032727d3ca53b3276b4769d178

In comments from https://review.openstack.org/8798 it was raised that it
might make sense to forbid some substrings in the name_filter
middleware.
There is now a new forbidden_regexp option for the name_filter
middleware to specify which substrings to forbid. The default is
"/\./|/\.\./|/\.$|/\.\.$" (or in a non-regexp language: the /./ and /../
substrings as well as strings ending with /. or /..).
This can be useful for extra paranoia to avoid directory traversals
(bug 1005908), or for more general filtering.
Change-Id: I39bf2de45b9dc7d3ca4d350d24b3f2276e958a62
DocImpact: new forbidden_regexp option for the name_filter middleware

We're still using saio:11000 in a few spots so a few things
don't work out of the box on the saio. Fixes bug #1024561
Change-Id: I226de54c2785b0d0b681c8d0cc24260adbd3d663

- Rename it to keystoneauth for consistenties.
- Implements blueprint keystone-middleware.
Change-Id: I208fecdf3ee991694b4239f065032324d297fd35

The documentation could be understood like the following:
memcache_servers from memcache.conf is always used if set, even if
memcache_servers in proxy-server.conf is set.
This is clearly not the case, as proxy-server.conf has a higher priority
if it memcache_servers is set there.
Change-Id: I967c7e80796a0e296c5c65bd097df1669d16203e