openstack/swift - swift - OpenDev: Free Software Needs Free Tools

The documentation could be understood like the following:
memcache_servers from memcache.conf is always used if set, even if
memcache_servers in proxy-server.conf is set.
This is clearly not the case, as proxy-server.conf has a higher priority
if it memcache_servers is set there.
Change-Id: I967c7e80796a0e296c5c65bd097df1669d16203e

Fixes LP Bug #1016656.
Change-Id: I22455d27cb82830dabd4fb0677da26915a60eb85

* Add new configuration option log_address.
Change-Id: I636bd4116687629c997b70a0d804b7ed4bc46032

The object server had a constant KEEP_CACHE_SIZE = 5*1024*1024;
unauthenticated GET requests for files smaller than KEEP_CACHE_SIZE
would not evict the file from the kernel's buffer cache after it was
read from disk.
Now that hardcoded constant is a configuration parameter
("keep_cache_size"), and now there is also another parameter called
"keep_cache_private". If set, then both authenticated and
unauthenticated GET requests for small files will not evict the data
from the buffer cache. The default values are 5 MiB and False,
respectively, so the default behavior is the same.
Bonus: the "mb_per_sync" parameter is now documented in the deployment
guide.
Change-Id: I9a11dbe861f4c23535c6aa82a9111a6fe2db2a59

bug 786197
Change-Id: Idd73986ca4550b20dff340df4acd12f3bfc01075

- It has been to its own gerrit project.
- direct_client should follow next.
- Implements blueprint clientbindings.
Change-Id: I3bb50c95eba81302bfec71cb7ce5288b85a41dc0

Change-Id: I319de933ecfb1e3853e3064656968c36980ce5f5

Change-Id: I771c87207d4e1821e32c3424b341d182cc7ea7c0

Expand recon middleware to include support for account and container
servers in addition to the existing object servers. Also add support
for retrieving recent information from auditors, replicators, and
updaters. In the case of certain checks (such as container auditors)
the stats returned are only for the most recent path processed.
The middleware has also been refactored and should now also handle
errors better in cases where stats are unavailable.
While new check's have been added the output from pre-existing
check's has not changed. This should allow existing 3rd party
utilities such as the Swift ZenPack to continue to function.
Change-Id: Ib9893a77b9b8a2f03179f2a73639bc4a6e264df7

- Reference https://github.com/fujita/swift3 in associated_projects.
- Implements blueprint add-associated-projects-docs.
Change-Id: I48ef4c03449edf6ef4fda1a391228cacac7d2ac6

Revert "removed cname lookup middleware"
This reverts commit b47bcf19e4.
Revert "removed domain remap middleware"
This reverts commit 317cf868bdf66dbc17c02d4ca383efafa5e2f229.
Change-Id: I260498d555c93b28896ace48a6f0e96701cbcc38

Long explanation, but hopefully answers any questions.
We don't like changing the default behavior of Swift unless there's a
really good reason and, up until now, I've tried doing this with this
new db_preallocation setting.
For clusters with dedicated account/container servers that usually
have fewer disks overall but SSD for speed, having db_preallocation
on will gobble up disk space quite quickly and the fragmentation it's
designed to fight isn't that big a speed impact to SSDs anyway.
For clusters with account/container servers spread across all servers
along with object servers usually having standard disks for cost,
having db_preallocation off will cause very fragmented database files
impacting speed, sometimes dramatically.
Weighing these two negatives, it seems the second is the lesser evil.
The first can cause disks to fill up and disable the cluster. The
second will cause performance degradation, but the cluster will still
function.
Furthermore, if just one piece of code that touches all databases
runs with db_preallocation on, it's effectively on for the whole
cluster. We discovered this most recently when we finally configured
everything within the Swift codebase to have db_preallocation off,
only to find out Slogging didn't know about the new setting and so
ran with it on and starting filling up SSDs.
So that's why I'm proposing this change to the default behavior.
We will definitely need to post a prominent notice of this change
with the next release.
Change-Id: I48a43439264cff5d03c14ec8787f718ee44e78ea

RateLimit
StaticWeb
TempURL/FormPOST
Change-Id: I988e93e6f4aacb817a2e354d43a04e47516fdf88

Documentation, including a list of metrics reported and their semantics,
is in the Admin Guide in a new section, "Reporting Metrics to StatsD".
An optional "metric prefix" may be configured which will be prepended to
every metric name sent to StatsD.
Here is the rationale for doing a deep integration like this versus only
sending metrics to StatsD in middleware. It's the only way to report
some internal activities of Swift in a real-time manner. So to have one
way of reporting to StatsD and one place/style of configuration, even
some things (like, say, timing of PUT requests into the proxy-server)
which could be logged via middleware are consistently logged the same
way (deep integration via the logger delegate methods).
When log_statsd_host is configured, get_logger() injects a
swift.common.utils.StatsdClient object into the logger as
logger.statsd_client. Then a set of delegate methods on LogAdapter
either pass through to the StatsdClient object or become no-ops. This
allows StatsD logging to look like:
 self.logger.increment('some.metric.here')
and do the right thing in all cases and with no messy conditional logic.
I wanted to use the pystatsd module for the StatsD client, but the
version on PyPi is lagging the git repo (and is missing both the prefix
functionality and timing_since() method). So I wrote my
swift.common.utils.StatsdClient. The interface is the same as
pystatsd.Client, but the code was written from scratch. It's pretty
simple, and the tests I added cover it. This also frees Swift from an
optional dependency on the pystatsd module, making this feature easier
to enable.
There's test coverage for the new code and all existing tests continue
to pass.
Refactored out _one_audit_pass() method in swift/account/auditor.py and
swift/container/auditor.py.
Fixed some misc. PEP8 violations.
Misc test cleanups and refactorings (particularly the way "fake logging"
is handled).
Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2

This commit introduces a new algorithm for assigning partition
replicas to devices. Basically, the ring builder organizes the devices
into tiers (first zone, then IP/port, then device ID). When placing a
replica, the ring builder looks for the emptiest device (biggest
parts_wanted) in the furthest-away tier.
In the case where zone-count >= replica-count, the new algorithm will
give the same results as the one it replaces. Thus, no migration is
needed.
In the case where zone-count < replica-count, the new algorithm
behaves differently from the old algorithm. The new algorithm will
distribute things evenly at each tier so that the replication is as
high-quality as possible, given the circumstances. The old algorithm
would just crash, so again, no migration is needed.
Handoffs have also been updated to use the new algorithm. When
generating handoff nodes, first the ring looks for nodes in other
zones, then other ips/ports, then any other drive. The first handoff
nodes (the ones in other zones) will be the same as before; this
commit just extends the list of handoff nodes.
The proxy server and replicators have been altered to avoid looking at
the ring's replica count directly. Previously, with a replica count of
C, RingData.get_nodes() and RingData.get_part_nodes() would return
lists of length C, so some other code used the replica count when it
needed the number of nodes. If two of a partition's replicas are on
the same device (e.g. with 3 replicas, 2 devices), then that
assumption is no longer true. Fortunately, all the proxy server and
replicators really needed was the number of nodes returned, which they
already had. (Bonus: now the only code that mentions replica_count
directly is in the ring and the ring builder.)
Change-Id: Iba2929edfc6ece89791890d0635d4763d821a3aa

The code has moved to https://github.com/notmyname/swift-cnamelookup.
For current users of cname lookup, this will require installing the new
package and changing the "use" line of the cname lookup conf section's
to:
[filter:cname_lookup]
use = egg:swift_cnamelookup#swift_cnamelookup
And then 'swift-init proxy reload'.
Change-Id: If622486ddb04a53251244c9840aa3cfe72168fc5

Rate Limit middleware is now at
http://dpgoetz.github.com/swift-ratelimit/
For current users of Rate Limit, this will require installing the new
package and changing the "use" line of the ratelimit conf section to:
[filter:ratelimit]
use = egg:swiftratelimit#middleware
And then 'swift-init proxy reload'.
Change-Id: I2ab774e9cee9fba4103c1be4bea6d52d1adb29f7

The code has moved to https://github.com/notmyname/swift-domainremap.
For current users of domain remap, this will require installing the new
package and changing the "use" line of the domain remap conf section's
to:
[filter:domain_remap]
use = egg:swift_domainremap#swift_domainremap
And then 'swift-init proxy reload'.
Change-Id: I710caf9b991f9d37df36b826ae4338086d0ec36d

TempURL/FormPOST is now at http://gholt.github.com/swift-tempurl/
For current users of TempURL/FormPOST, this will require installing the new
package and changing the "use" line of the tempurl and formpost conf section's
to:
[filter:tempurl]
use = egg:swifttempurl#tempurl
[filter:formpost]
use = egg:swifttempurl#formpost
And then 'swift-init proxy reload'.
Change-Id: I5bddf7f9e09ee07815530a41c46ff901fc21b447

StaticWeb is now at http://gholt.github.com/swift-staticweb/
For current users of StaticWeb, this will require installing the new
package and changing the "use" line of the staticweb filter conf
section to:
use = egg:swiftstaticweb#middleware
And then 'swift-init proxy reload'.
Change-Id: Iab32adb5927698a667c5c6d6a572c44ca23414eb

Refactored object expirer to use this client.
Change-Id: Ibeca6dba873f8b4a558ecf3ba6e8d23d36f545b0

In the interest of keeping the core Swift code repository less
complex, but still offering a quick way to find associated projects
that enhance or use Swift, I've added this new Associated Projects
page prominently to the Swift documentation.
This will allow much less barrier to entry on enhancing Swift as
other projects can work independently and will only need to wait on
the core Swift project for approval of minimal tweaks to the core
Swift code base.
It will also allow an easy central place to find cool associated
projects that otherwise might go unnoticed or even duplicated.
The plan is to move non-essential projects that are currently
embedded in the Swift repository out into their own projects with
links to them on this new page. This would include items such as
(just what I can think of right now): bin/swift command line tool and
clients, swift-bench, swift-dispersion, TempURL, FormPost, StaticWeb,
Rate Limiting, Swift3, Domain Remap, and CNAME Lookup.
After all that is done, those projects will be able to move forward
much more quickly and new developers for Swift itself will have much
less to learn and get confused about.
Change-Id: Ib8447d8bd480f0a3d8f0413137ccdba73a11dd91

bug 701248
bug 819303
This change makes syslog optional, by providing a unit test
config with the ability to cause test/unit/__init__.py to
replace SysLogHandler with a fake logging class. The
default behavior is unchanged.
FakeLogger now inherits directly from Handler and mocks out
its API - this was a backward-compatibility issue.
Change-Id: I653dec148c4f6cf81759de03964c6a3290c1a290

Change-Id: Idf2ea227b0078026eb2693ff7c0a24ec926bb6f3

Change-Id: I766b2da0a3769869b5b6c5565106e9032fea8c3e

In the side bar, links to past versions and docs.openstack.org
On the index.rst, links to wiki and docs.openstack.org
Change-Id: Icf33c6f396e1ab016fd86a56e61df3e063a1bae2

Change-Id: Id68231ced070b27d1f7fb5a230d345694632e7dd

Corrected its/it's mistakes, harmonized line wrapping within some docs
and clarified doc wording in several places.
Change-Id: Ib9ac6d5e859f770a702e1fad6de8d4abe0390b47

Admittedly, this is ridiculously trivial, but the coloring of
"hash" always peeved me. Not so much that it existed, but that
I did not understand what caused it. After consulting our Sphinx
experts, it turned out that the code block was considered code
in Python and "hash" is a reserved word in Python, so it was
colored.
One option is to declare the code block "ini", which is what
it is, but that causes the block stand out just as badly, unless
we highlight every one of them. So, for now use "none".
Change-Id: Icee021fc30839eeb02cd3e232ff6be92ed32df52

Change-Id: I1c571951f25a6e724cdd4699eb94baad5e47eb95

Added option to disable SQLite db preallocation. This can be very
useful on pure ssd account/container servers where the extra space is
worth more than the lesser fragmentation.
Change-Id: I8fbb028a9b6143775b25b343e97896497a8b63a9

Object versioning in swift is implemented by setting a flag on the container
to tell swift to version all objects in the container. The flag is the
``X-Versions-Location`` header on the container, and its value is the
container where the versions are stored.
When data is ``PUT`` into a versioned container (a container with the
versioning flag turned on), the existing data in the file is redirected to a
new object and the data in the ``PUT`` request is saved as the data for the
versioned object. The new object name (for the previous version) is
``<versions_container>/<object_name>/<timestamp>``, where the timestamp is
generated by converting the ``Last-Modified`` header value of the current
version to a unix timestamp.
A ``GET`` to a versioned object will return the current version of the object
without having to do any request redirects or metadata lookups.
Change-Id: I4fcd723145e02bbb2ec1d3ad356713f5dea43b8b

Normally, the reaper begins deleting account information for deleted
accounts immediately. With this patch you can set it to delay its
work. You set the delay_reaping value in the [account-reaper] section
of the account-server.conf. The value is in seconds; 2592000 = 30
days, for example.
Unfortunately, there are currently zero tests for the account-reaper.
This also needs fixing, but I thought I'd submit this delay patch
alone for consideration.
Change-Id: Ic077df9cdd95c5d3f8949dd3bbe9893cf24c6623

Change-Id: Ifd909d3561c2647770a7e0caa3cd91acd1b4f298

Added optional max_containers_per_account restriction. If set to a
positive value and if a client tries to perform a container PUT when
at or above the max_containers_per_acount cap, a 403 Forbidden will
be returned with an explanatory message.
This only restricts the proxy server, not any of the background
processes that might need to create containers (replication, for
instance). Also, the container count is cached for the proxy's
recheck_account_existence number of seconds. For these reasons, a
given account could exceed this cap before the 403 Forbidden
responses kick in and therefore this feature should be considered a
"soft" limit.
You may also add accounts to the proxy's max_containers_whitelist
setting to have accounts that ignore this cap.
Change-Id: I74e8fb152de5e78d070ed30006ad4e53f82c8376

Change-Id: I081d853a90ad0efce5b7a20b52794ce3348a9617