openstack/swift - swift - OpenDev: Free Software Needs Free Tools

A deployer may want to remove a Swift node from a load balancer for
maintenance or upgrade. This patch provides an optional mechanism for
this. The healthcheck filter config can specify "disable_path" which is
a filesystem path. If a file is present at that location, the
healthcheck middleware returns a 503 with a body of "DISABLED BY FILE".
So a deployer can configure "disable_path" and then touch that
filesystem path, wait for the proxy to be removed from the load balancer
pool, perform maintenance/upgrade, and then remove the "disable_path"
file.
Also cleaned up the conf file man pages a bit.
Change-Id: I1759c78c74910a54c720f298d4d8e6fa57a4dab4

Updates the proxy-server.conf-sample and docs to use
the new Keystoneclient middleware class name.
Change-Id: I3727f7b7328a2513347b8ef257c270126df36d7b

I know it's just TempAuth, but bug #959953 just caught my eye as
something interesting to solve.
This does a best guess on the storage URL to return for a given
request. It allows $HOST to be used in the storage URL configuration,
where $HOST will resolve to scheme://host:port. It bases the scheme
on how the server is running or on storage_url_scheme if set. The
host:port comes from the request's Host header if it exists, and
falls back to the WSGI SERVER_NAME:SERVER_PORT otherwise.
Fixes: bug #959953
DocImpact
Change-Id: Ia494bcb99a04490911ee8d2cb8b12a94e77820c5

Change-Id: I77e1d7fdcf217826402beeb7d583e3c7279c416c

Change-Id: I894473994cdfea0996ad16e7619aff421f604abc

Add a hook to get_logger to run custom functions to add custom log
handlers or the like.
Change-Id: Ib04b12939dcac7e4ad6453dea9795682044c6ae0

This patch introduces a new user format to tempauth which starts with
"user64_" and in which the account and user are base64 encoded (sans
padding equal signs):
user64_<account_b64>_<user_b64> = <key> [group] [group] [...] [storage_url]
This allows unambiguous parsing and adds the ability to have a user or
account name containing underscores.
The handling of tempauth configuration is fully backward compatible and
this only adds new, optional functionality.
Also beefed up unit tests to get full coverage of the user parsing code
in tempauth's __init__().
Change-Id: Id06304934ab8055a28921f2e1aa3c58a2036d8f8

Change previously hard-coded constants into config variables. This
allows deployers to tune their cluster more specifically based on
their needs. For example, a deployment that uses direct swift access
for public content may need to set a larger header value constraint to
allow for the full object name to be represented in the Content-
Disposition header (which browsers check to determine the name of a
downloaded object).
The new settings are set in the [swift-constraints] section of
/etc/swift/swift.conf. Comments were also added to this config file.
Cleaned up swift/common/constraints.py to pass pep8 1.3.3
Funtional tests now require constraints to be defined in /etc/test.conf or in
/etc/swift/swift.conf (in the case of running the functional tests against a
local swift cluster). To have any hope of tests passing, the defined
constraints must match the constraints on the tested cluster.
Removed a ton of "magic numbers" in both unit and functional tests.
Change-Id: Ie4588e052fd158314ddca6cd8fca9bc793311465

Removed many StatsD logging calls in proxy-server and added
swift-informant-style catch-all logging in the proxy-logger middleware.
Many errors previously rolled into the "proxy-server.<type>.errors"
counter will now appear broken down by response code and with timing
data at: "proxy-server.<type>.<verb>.<status>.timing". Also, bytes
transferred (sum of in + out) will be at:
"proxy-server.<type>.<verb>.<status>.xfer". The proxy-logging
middleware can get its StatsD config from standard vars in [DEFAULT] or
from access_log_statsd_* config vars in its config section.
Similarly to Swift Informant, request methods ("verbs") are filtered
using the new proxy-logging config var, "log_statsd_valid_http_methods"
which defaults to GET, HEAD, POST, PUT, DELETE, and COPY. Requests with
methods not in this list use "BAD_METHOD" for <verb> in the metric name.
To avoid user error, access_log_statsd_valid_http_methods is also
accepted.
Previously, proxy-server metrics used "Account", "Container", and
"Object" for the <type>, but these are now all lowercase.
Updated the admin guide's StatsD docs to reflect the above changes and
also include the "proxy-server.<type>.handoff_count" and
"proxy-server.<type>.handoff_all_count" metrics.
The proxy server now saves off the original req.method and proxy_logging
will use this if it can (both for request logging and as the "<verb>" in
the statsd timing metric). This fixes bug 1025433.
Removed some stale access_log_* related code in proxy/server.py. Also
removed the BaseApplication/Application distinction as it's no longer
necessary.
Fixed up the sample config files a bit (logging lines, mostly).
Fixed typo in SAIO development guide.
Got proxy_logging.py test coverage to 100%.
Fixed proxy_logging.py for PEP8 v1.3.2.
Enhanced test.unit.FakeLogger to track more calls to enable testing
StatsD metric calls.
Change-Id: I45d94cb76450be96d66fcfab56359bdfdc3a2576

Change-Id: Id8872c581ed23378a8e14cbf3bf049b5c0d21577

swift-bench now honors the environment variables, ST_AUTH, ST_USER, and
ST_KEY like python-swiftclient does.
Added --lower-object-size (or -l) command-line option which, if
specified, will turn a specified --object-size into --upper-object-size.
It will raise a ValueError if --object-size is not specified or is <=
--lower-object-size.
BenchController how handles SIGINT (KeyboardInterrupt) similarly to the
swift command-line client: the first Ctrl-C will make it fast-track to
completion (no new PUT or GET operations are started, but everything PUT
is DELETE'ed). A second Ctrl-C will immediately exit. The behavior for
SIGTERM is unchanged (a single SIGTERM will immediately terminate the
process).
Added a sample configuration file for swift-bench, with documenting
comments.
Change-Id: I6f394ad995300fc8af3d565d95c3b45559ee510a

If rate_limit_after_segment was 10 in the proxy config, then after 10
segments were coughed up by _load_next_segment() with no calls to
sleep(), the 11th segment would not trigger a sleep() call. The 12th
segment triggered a sleep(0) call, but it was only after the 13th
segment was loaded that an actual rate-limiting (non-zero) sleep got
called.
With this patch, a rate_limit_after_segment of 10 will start sleeping
the correct amount after the 11th segment.
Updated proxy-server.conf-sample with rate_limit_after_segment and
rate_limit_segments_per_sec.
Change-Id: I937c366996e6d6ab47c614d6db470e3be9657c07

We don't want to use pickle as it can execute arbitrary code. JSON is
safer. However, note that it supports serialization for only some
specific subset of object types; this should be enough for what we need,
though.
To avoid issues on upgrades (unability to read pickled values, and cache
poisoning for old servers not understanding JSON), we add a
memcache_serialization_support configuration option, with the following
values:
 0 = older, insecure pickle serialization
 1 = json serialization but pickles can still be read (still insecure)
 2 = json serialization only (secure and the default)
To avoid an instant full cache flush, existing installations should
upgrade with 0, then set to 1 and reload, then after some time (24
hours) set to 2 and reload. Support for 0 and 1 will be removed in
future versions.
Part of bug 1006414.
Change-Id: Id7d6d547b103b4f23ebf5be98b88f09ec6027ce4

A warning log line is emitted whenever the proxy has to use a handoff
node. Monitoring these warnings can indicate a problem within your
cluster; however, you can disable these log lines by setting the
proxy conf's log_handoffs to false.
While working on this, I also noticed why many proxy log lines did
not have txn_id and client_ip -- subcoroutines. Now the logger thread
locals are copied to the subcoroutines.
Change-Id: Ibac086e1b985f566c068d083620287509de35da8

In comments from https://review.openstack.org/8798 it was raised that it
might make sense to forbid some substrings in the name_filter
middleware.
There is now a new forbidden_regexp option for the name_filter
middleware to specify which substrings to forbid. The default is
"/\./|/\.\./|/\.$|/\.\.$" (or in a non-regexp language: the /./ and /../
substrings as well as strings ending with /. or /..).
This can be useful for extra paranoia to avoid directory traversals
(bug 1005908), or for more general filtering.
Change-Id: I39bf2de45b9dc7d3ca4d350d24b3f2276e958a62
DocImpact: new forbidden_regexp option for the name_filter middleware

We're still using saio:11000 in a few spots so a few things
don't work out of the box on the saio. Fixes bug #1024561
Change-Id: I226de54c2785b0d0b681c8d0cc24260adbd3d663

- Rename it to keystoneauth for consistenties.
- Implements blueprint keystone-middleware.
Change-Id: I208fecdf3ee991694b4239f065032324d297fd35

The documentation could be understood like the following:
memcache_servers from memcache.conf is always used if set, even if
memcache_servers in proxy-server.conf is set.
This is clearly not the case, as proxy-server.conf has a higher priority
if it memcache_servers is set there.
Change-Id: I967c7e80796a0e296c5c65bd097df1669d16203e

* Add new configuration option log_address.
Change-Id: I636bd4116687629c997b70a0d804b7ed4bc46032

The object server had a constant KEEP_CACHE_SIZE = 5*1024*1024;
unauthenticated GET requests for files smaller than KEEP_CACHE_SIZE
would not evict the file from the kernel's buffer cache after it was
read from disk.
Now that hardcoded constant is a configuration parameter
("keep_cache_size"), and now there is also another parameter called
"keep_cache_private". If set, then both authenticated and
unauthenticated GET requests for small files will not evict the data
from the buffer cache. The default values are 5 MiB and False,
respectively, so the default behavior is the same.
Bonus: the "mb_per_sync" parameter is now documented in the deployment
guide.
Change-Id: I9a11dbe861f4c23535c6aa82a9111a6fe2db2a59

Change-Id: I771c87207d4e1821e32c3424b341d182cc7ea7c0

Expand recon middleware to include support for account and container
servers in addition to the existing object servers. Also add support
for retrieving recent information from auditors, replicators, and
updaters. In the case of certain checks (such as container auditors)
the stats returned are only for the most recent path processed.
The middleware has also been refactored and should now also handle
errors better in cases where stats are unavailable.
While new check's have been added the output from pre-existing
check's has not changed. This should allow existing 3rd party
utilities such as the Swift ZenPack to continue to function.
Change-Id: Ib9893a77b9b8a2f03179f2a73639bc4a6e264df7

Revert "removed cname lookup middleware"
This reverts commit b47bcf19e4.
Revert "removed domain remap middleware"
This reverts commit 317cf868bdf66dbc17c02d4ca383efafa5e2f229.
Change-Id: I260498d555c93b28896ace48a6f0e96701cbcc38

Long explanation, but hopefully answers any questions.
We don't like changing the default behavior of Swift unless there's a
really good reason and, up until now, I've tried doing this with this
new db_preallocation setting.
For clusters with dedicated account/container servers that usually
have fewer disks overall but SSD for speed, having db_preallocation
on will gobble up disk space quite quickly and the fragmentation it's
designed to fight isn't that big a speed impact to SSDs anyway.
For clusters with account/container servers spread across all servers
along with object servers usually having standard disks for cost,
having db_preallocation off will cause very fragmented database files
impacting speed, sometimes dramatically.
Weighing these two negatives, it seems the second is the lesser evil.
The first can cause disks to fill up and disable the cluster. The
second will cause performance degradation, but the cluster will still
function.
Furthermore, if just one piece of code that touches all databases
runs with db_preallocation on, it's effectively on for the whole
cluster. We discovered this most recently when we finally configured
everything within the Swift codebase to have db_preallocation off,
only to find out Slogging didn't know about the new setting and so
ran with it on and starting filling up SSDs.
So that's why I'm proposing this change to the default behavior.
We will definitely need to post a prominent notice of this change
with the next release.
Change-Id: I48a43439264cff5d03c14ec8787f718ee44e78ea

RateLimit
StaticWeb
TempURL/FormPOST
Change-Id: I988e93e6f4aacb817a2e354d43a04e47516fdf88

Documentation, including a list of metrics reported and their semantics,
is in the Admin Guide in a new section, "Reporting Metrics to StatsD".
An optional "metric prefix" may be configured which will be prepended to
every metric name sent to StatsD.
Here is the rationale for doing a deep integration like this versus only
sending metrics to StatsD in middleware. It's the only way to report
some internal activities of Swift in a real-time manner. So to have one
way of reporting to StatsD and one place/style of configuration, even
some things (like, say, timing of PUT requests into the proxy-server)
which could be logged via middleware are consistently logged the same
way (deep integration via the logger delegate methods).
When log_statsd_host is configured, get_logger() injects a
swift.common.utils.StatsdClient object into the logger as
logger.statsd_client. Then a set of delegate methods on LogAdapter
either pass through to the StatsdClient object or become no-ops. This
allows StatsD logging to look like:
 self.logger.increment('some.metric.here')
and do the right thing in all cases and with no messy conditional logic.
I wanted to use the pystatsd module for the StatsD client, but the
version on PyPi is lagging the git repo (and is missing both the prefix
functionality and timing_since() method). So I wrote my
swift.common.utils.StatsdClient. The interface is the same as
pystatsd.Client, but the code was written from scratch. It's pretty
simple, and the tests I added cover it. This also frees Swift from an
optional dependency on the pystatsd module, making this feature easier
to enable.
There's test coverage for the new code and all existing tests continue
to pass.
Refactored out _one_audit_pass() method in swift/account/auditor.py and
swift/container/auditor.py.
Fixed some misc. PEP8 violations.
Misc test cleanups and refactorings (particularly the way "fake logging"
is handled).
Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2

The code has moved to https://github.com/notmyname/swift-cnamelookup.
For current users of cname lookup, this will require installing the new
package and changing the "use" line of the cname lookup conf section's
to:
[filter:cname_lookup]
use = egg:swift_cnamelookup#swift_cnamelookup
And then 'swift-init proxy reload'.
Change-Id: If622486ddb04a53251244c9840aa3cfe72168fc5

Rate Limit middleware is now at
http://dpgoetz.github.com/swift-ratelimit/
For current users of Rate Limit, this will require installing the new
package and changing the "use" line of the ratelimit conf section to:
[filter:ratelimit]
use = egg:swiftratelimit#middleware
And then 'swift-init proxy reload'.
Change-Id: I2ab774e9cee9fba4103c1be4bea6d52d1adb29f7

The code has moved to https://github.com/notmyname/swift-domainremap.
For current users of domain remap, this will require installing the new
package and changing the "use" line of the domain remap conf section's
to:
[filter:domain_remap]
use = egg:swift_domainremap#swift_domainremap
And then 'swift-init proxy reload'.
Change-Id: I710caf9b991f9d37df36b826ae4338086d0ec36d

TempURL/FormPOST is now at http://gholt.github.com/swift-tempurl/
For current users of TempURL/FormPOST, this will require installing the new
package and changing the "use" line of the tempurl and formpost conf section's
to:
[filter:tempurl]
use = egg:swifttempurl#tempurl
[filter:formpost]
use = egg:swifttempurl#formpost
And then 'swift-init proxy reload'.
Change-Id: I5bddf7f9e09ee07815530a41c46ff901fc21b447

StaticWeb is now at http://gholt.github.com/swift-staticweb/
For current users of StaticWeb, this will require installing the new
package and changing the "use" line of the staticweb filter conf
section to:
use = egg:swiftstaticweb#middleware
And then 'swift-init proxy reload'.
Change-Id: Iab32adb5927698a667c5c6d6a572c44ca23414eb

Added support for configuring lock file location.
added an additional error handler.
Change-Id: I3635ed9694b9fb8526e0ca0591dc731da03a7128

Change-Id: I41356ee250c9088a2387b0d493586dd990a04ac3

Change-Id: I766b2da0a3769869b5b6c5565106e9032fea8c3e

Change-Id: I974f729da60e5ab9453daf9e52466b3e1af5c69b

Change-Id: I080d531471d8ea57c69a918d4f6930441f1e69f6
Signed-off-by: Julien Danjou <julien.danjou@enovance.com>

Added option to disable SQLite db preallocation. This can be very
useful on pure ssd account/container servers where the extra space is
worth more than the lesser fragmentation.
Change-Id: I8fbb028a9b6143775b25b343e97896497a8b63a9

Object versioning in swift is implemented by setting a flag on the container
to tell swift to version all objects in the container. The flag is the
``X-Versions-Location`` header on the container, and its value is the
container where the versions are stored.
When data is ``PUT`` into a versioned container (a container with the
versioning flag turned on), the existing data in the file is redirected to a
new object and the data in the ``PUT`` request is saved as the data for the
versioned object. The new object name (for the previous version) is
``<versions_container>/<object_name>/<timestamp>``, where the timestamp is
generated by converting the ``Last-Modified`` header value of the current
version to a unix timestamp.
A ``GET`` to a versioned object will return the current version of the object
without having to do any request redirects or metadata lookups.
Change-Id: I4fcd723145e02bbb2ec1d3ad356713f5dea43b8b

Normally, the reaper begins deleting account information for deleted
accounts immediately. With this patch you can set it to delay its
work. You set the delay_reaping value in the [account-reaper] section
of the account-server.conf. The value is in seconds; 2592000 = 30
days, for example.
Unfortunately, there are currently zero tests for the account-reaper.
This also needs fixing, but I thought I'd submit this delay patch
alone for consideration.
Change-Id: Ic077df9cdd95c5d3f8949dd3bbe9893cf24c6623

Added optional max_containers_per_account restriction. If set to a
positive value and if a client tries to perform a container PUT when
at or above the max_containers_per_acount cap, a 403 Forbidden will
be returned with an explanatory message.
This only restricts the proxy server, not any of the background
processes that might need to create containers (replication, for
instance). Also, the container count is cached for the proxy's
recheck_account_existence number of seconds. For these reasons, a
given account could exceed this cap before the 403 Forbidden
responses kick in and therefore this feature should be considered a
"soft" limit.
You may also add accounts to the proxy's max_containers_whitelist
setting to have accounts that ignore this cap.
Change-Id: I74e8fb152de5e78d070ed30006ad4e53f82c8376

Bug 926048.
Filter checks path for user-defined forbidden characters, and for
user-defined maximum length.
Includes changes to reflect gholt's latest comments to Patch Set 4
Also includes a change to a unit-test, renames another unit-test,
and removes one superfluous unit-test.
Added section to the example proxy config
Fixed-up unit test pep8 warnings
Changed error response code to 400 (Bad Request)
Change-Id: Iace719d6a3d00fb3dda1b9d0bc185b8c4cbc00ca

https://blueprints.launchpad.net/swift/+spec/remove-old-swift-stats-tools
Change-Id: Ibb57b3d71c7f18060d2d48a1d3754a7f1444624d

Note: I'd like to get this released as soon as possible as it is a
data durability issue.
1) Orders nodes so that none get starved (see code and footnote).
2) New max_diffs setting that caps how long the replicator will spend
 trying to sync a given database per pass so the other databases
 don't get starved.
3) Replaces run_pause with the more standard "interval", which means
 the replicator won't pause unless it takes less than the interval
 set.
Change-Id: I986742229e65031df88f5251ca61746b7c8d2bde

Add's the configuration file option "dump_json" or command line
options [-j|--dump-json] to have swift-dispersion-report output
the report in json format. This allows the dispersion report to
be more easily consumed elsewhere.
There's also a few pep8 fixes and removal of unused imports.
Change-Id: I2374311ccbef43e6bbae24665c9584e60f3da173