openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Add unit tests to verify the precedence of access_log_ and log_
prefixes to options.
Add pointers from proxy_logging sections in other sample config files
to the proxy-server.conf-sample file.
Change-Id: Id18176d3790fd187e304f0e33e3f74a94dc5305c

Add an example of a delay_reaping config option with quoted key.
Change-Id: I0c7ead6795822ea0fb0e81abc1e4685d7946942c
Related-Change: I106103438c4162a561486ac73a09436e998ae1f0

The object expirer can be configured to delay the reaping of
objects from disk after their expiration time using account
and container level delay_reaping values. The delay_reaping
value of accounts and containers in seconds is configured in
the object server config. The object expirer references these
configured values to only reap objects from specified accounts
and containers after their corresponding delays.
The goal of the delay_reaping feature is to prevent accidental or
premature data loss if an object marked for deletion with the
'x-delete-at' feature should not be reaped immediately, for
whatever reason.
Configuring the delay_reaping value at a granular account and
container level is beneficial for being able to keep storage
capacity consumption in control while maintaining a desired
data recovery window.
This patch also adds a sample configuration, documentation, and
tests for bad configurations and grace period functionality.
Co-Authored-By: Anish Kachinthaya <akachinthaya@nvidia.com>
Change-Id: I106103438c4162a561486ac73a09436e998ae1f0

Change-Id: I91e9bc02d94fe7ea6e89307305705c383087845a

This allows operators to throttle expirers as needed.
Partial-Bug: #1784753
Change-Id: If75dabb431bddd4ad6100e41395bb6c31a4ce569

If we move it to constraints it's more globally accessible in our code,
but more importantly it's more obvious to ops that everything breaks if
you try to mis-configure different values per-service.
Change-Id: Ib8f7d08bc48da12be5671abe91a17ae2b49ecfee

...and update the object-expirer man page.
Change-Id: Idca1b8e3b7d5b40481af0d60477510e2557b88c0

Based on comments from patch #238799.
Change-Id: I9455cf6dc7fd12fee62439ff3c5f3255287ab1be

The goal is to modify schedule priority and I/O scheduling class and
priority of daemon/server via configuration.
Setting is optional, default keeps current behaviour.
Use case:
Prioritize object-server to object-auditor, because all user's requests
needed to be served in peak hours and audit could wait.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
DocImpact
Change-Id: I1018a18f4706daabdb84574ffd9a58d831e68396

Multiple files and documents showed that log_statsd_host had
a default value, usually localhost. This was incorrect, instead
setting a value for log_statsd_host enables statsd logging.
Removed any reference of log_statsd_host having a default value.
Also changed descriptions to show setting a value enables logging.
Change-Id: I3ca5c0e8b8e4981de3aa6db0c476072b5a59723d
Closes-Bug: #1542227 

s/overide/override for object-expirer.conf and sample.
s/automaticaly/automatically for swift/proxy/controllers/obj.py
Change-Id: Ife107c7a1005a5d4959288db50a7f8f33c522dd4
Signed-off-by: Zhao Lei <zhaolei@cn.fujitsu.com>

This change is the result of an audit through the config parameters
provided by swift and how/if they are addressed in the swift
documentation. The documentation being the sample config files in
the /etc directory or the documentation.
This change is only concerned with the config files in etc/ next
I will look at the documentation in the doc/ folder.
This change makes the following assumptions:
 - Unless stated otherwise, the commented out parameter in the
 sample configuration is the default for swift.
 - When the default in the code differs from that of the sample
 configuration, the default in the code is correct.
Container reconciler:
 Parameter: interval
 - code: 30
 - config: 300
 Result: config = 30
Object Expirer:
 Parameter: recon_cache_path
 - code: /var/cache/swift
 - config: Parameter missing
 Result: Add parameter
swift-dispersion-populate && swift-dispersion-report
 Parameter: auth_version
 - code: 1.0
 - config: 2.0 (due to being a confusing example of how to setup
 version 2.0).
 Result: Added 'auth_version = 1.0' to the right section (showing
 default and make the sample configuration for auth version
 2.0 easier to understand.
swift-drive-audit:
 Parameter: log_file_pattern
 - code: /var/log/kern.*[!.][!g][!z]
 - config: /var/log/kern*
 Result: config = /var/log/kern.*[!.][!g][!z]
 NOTE: swift-drive-audit uses a parameter called device_dir which
 defaults to '/srv/node'. In all other swift binaries/services
 there is a similar parameter called devices which stores the
 same thing. This is an inconsistency which I haven't fixed
 as this could break existing swift clusters out in the wild.
Proxy Server:
 Parameter: object_chunk_size
 - code: 65536
 - config: 8192
 Result: config = 65536
 Parameter: client_chunk_size
 - code: 65536
 - config: 8192
 Result: config = 65536
 Parameter: strict_cors_mode
 - code: True
 - config: No parameter
 Result: config = True
Account and Container replicator configuration confusion:
 NOTES:
 The account and container replicators have parameters:
 - interval
 - run_pause
 Both of these are loaded into the same variable in code:
 self.interval = int(conf.get('interval') or
 conf.get('run_pause') or 30)
 If a user sets both to different values then interval is used.
 Result: Update the configuration to make this more clear.
DocImpact
Change-Id: Iaadbb1a6284f8b3e0801bc343b29772f70f4bf6e

It's generally better to have logs for something than to not have
logs. This way, the object expirer (if using the sample config as a
starting point) will log what it does.
Note that the container reconciler's sample config already contains
proxy-logging, as does the proxy server's. The object expirer is the
odd man out.
Change-Id: I32aac99131746501820319b94405440c1934a694

Currently if the object-expirer goes to delete an object and the primary nodes
are unavailable, or the object is on handoffs - the object servers are unable
to verify the x-if-delete-at timestamp and return 412, without writing a
tombstone or updating the containers. The expirer treats 412 as success and
the dark data is not removed form the object servers nor the object removed in
the listing.
As a side effect of this bug, if the expirer encounters split brain the delete
would never get processed in the correct storage policy.
It seems it's just not correct to treat the lack of data as success. Now the
object server will treat x-if-delete at against a non-existent object as a
404, and to distinguish from a successfull process of an x-if-delete-at
request, will return 204.
The expirer will treat a 404 response from swift as a failure, and will
continue to attempt to expire the object until it is older that it's
configurable reclaim age. However swift will only return 404 if the majority
of nodes are able to return success, or if only even a single node is able to
accept the x-if-delete-at request the containers will get updated and
replicaiton will settle the tombstone - the subsequent x-if-delete-at request
will 412 and be removed from the queue.
It's worth noting that if an object with x-delete-at meta is DELETED (by a
client request) an async update for the expiring update containers will be
processed to remove the queue entry - but if no primary nodes handle the
DELETE request replication will never remove the expiring entry and assuming
it's scheduled for beyond the tombstones reclaim age - the queue entry will
not be processable. In this case the expirer will attempt to DELETE the
object (and get 404s) in vain until the queue entry passes the configurable
reclaim age.
DocImpact
Implements: blueprint storage-policies
Change-Id: I66260e99fda37e97d6d2470971b6f811ee9e01be

Log lines can get quite large, as we previously noticed with rsync error
log lines. We added a setting to cap those, but it really looks like we
should have just done this overall limit. We noticed the issue when we
switched to UDP syslogging and it would occasionally blow past the 16436
lo MTU! This causes Python's logging code to get an error and hilarity
ensues.
Change-Id: I44bdbe68babd58da58c14360379e8fef8a6b75f7

The account which tracks objects scheduled for deletion had its name
hard-coded to 'expiring_objects'. This is made configurable via
expiring_objects_account_name option. Backend file-systems
integration efforts may want to treat these "special" accounts in a
different way.
This would still go undocumented, hence 'pseudo-hidden'.
UpgradeImpact: None as the default value would continue to be the same
 which is '.expiring_objects'.
Change-Id: I1a093b0d0e2bdd0c3d723090af03fc0adf2ad7e3
Signed-off-by: Prashanth Pai <ppai@redhat.com>

Two types of parallelism are added:
- concurrency to speed up what a single process does
- a way to run multiple daemons to work on different parts of the work
DocImpact
Change-Id: I48997f68eb2fd8de19a5ee8b9fcdf76dde2ba0ab

Inject some empty lines to avoid the wall-of-text effect and to make
it a little clearer which descriptions go with which options.
Change-Id: I58914b83dad76ea5ca330903a246bee7ffaeba83

As Dieter pointed out in bug 1090495
(https://bugs.launchpad.net/swift/+bug/1090495), the volume of metrics
can vary wildly between StatsD metrics.
This patch implements a partial solution by reducing the sample_rate
used for known high-volume metrics (operational experience will need to
inform this over time) and introducing a new tunable,
log_statsd_sample_rate_factor which is multiplied by the sample_rate for
every statsd stat. This tunable can be used to reduce StatsD traffic
proportionally for all metrics and is intended to replace
log_statsd_default_sample_rate, which is left alone for
backward-compatibility, should anyone be using it.
This patch also includes a drive-by fix for log_udp_port which wasn't
being converted to an int (I didn't verify that actually causes trouble
in SysLogHandler(), but it's definitely an improvement regardles).
Change-Id: Id404636e3629f6431cf1c4e64a143959750a3c23

Add a hook to get_logger to run custom functions to add custom log
handlers or the like.
Change-Id: Ib04b12939dcac7e4ad6453dea9795682044c6ae0

Removed many StatsD logging calls in proxy-server and added
swift-informant-style catch-all logging in the proxy-logger middleware.
Many errors previously rolled into the "proxy-server.<type>.errors"
counter will now appear broken down by response code and with timing
data at: "proxy-server.<type>.<verb>.<status>.timing". Also, bytes
transferred (sum of in + out) will be at:
"proxy-server.<type>.<verb>.<status>.xfer". The proxy-logging
middleware can get its StatsD config from standard vars in [DEFAULT] or
from access_log_statsd_* config vars in its config section.
Similarly to Swift Informant, request methods ("verbs") are filtered
using the new proxy-logging config var, "log_statsd_valid_http_methods"
which defaults to GET, HEAD, POST, PUT, DELETE, and COPY. Requests with
methods not in this list use "BAD_METHOD" for <verb> in the metric name.
To avoid user error, access_log_statsd_valid_http_methods is also
accepted.
Previously, proxy-server metrics used "Account", "Container", and
"Object" for the <type>, but these are now all lowercase.
Updated the admin guide's StatsD docs to reflect the above changes and
also include the "proxy-server.<type>.handoff_count" and
"proxy-server.<type>.handoff_all_count" metrics.
The proxy server now saves off the original req.method and proxy_logging
will use this if it can (both for request logging and as the "<verb>" in
the statsd timing metric). This fixes bug 1025433.
Removed some stale access_log_* related code in proxy/server.py. Also
removed the BaseApplication/Application distinction as it's no longer
necessary.
Fixed up the sample config files a bit (logging lines, mostly).
Fixed typo in SAIO development guide.
Got proxy_logging.py test coverage to 100%.
Fixed proxy_logging.py for PEP8 v1.3.2.
Enhanced test.unit.FakeLogger to track more calls to enable testing
StatsD metric calls.
Change-Id: I45d94cb76450be96d66fcfab56359bdfdc3a2576

* Add new configuration option log_address.
Change-Id: I636bd4116687629c997b70a0d804b7ed4bc46032

Documentation, including a list of metrics reported and their semantics,
is in the Admin Guide in a new section, "Reporting Metrics to StatsD".
An optional "metric prefix" may be configured which will be prepended to
every metric name sent to StatsD.
Here is the rationale for doing a deep integration like this versus only
sending metrics to StatsD in middleware. It's the only way to report
some internal activities of Swift in a real-time manner. So to have one
way of reporting to StatsD and one place/style of configuration, even
some things (like, say, timing of PUT requests into the proxy-server)
which could be logged via middleware are consistently logged the same
way (deep integration via the logger delegate methods).
When log_statsd_host is configured, get_logger() injects a
swift.common.utils.StatsdClient object into the logger as
logger.statsd_client. Then a set of delegate methods on LogAdapter
either pass through to the StatsdClient object or become no-ops. This
allows StatsD logging to look like:
 self.logger.increment('some.metric.here')
and do the right thing in all cases and with no messy conditional logic.
I wanted to use the pystatsd module for the StatsD client, but the
version on PyPi is lagging the git repo (and is missing both the prefix
functionality and timing_since() method). So I wrote my
swift.common.utils.StatsdClient. The interface is the same as
pystatsd.Client, but the code was written from scratch. It's pretty
simple, and the tests I added cover it. This also frees Swift from an
optional dependency on the pystatsd module, making this feature easier
to enable.
There's test coverage for the new code and all existing tests continue
to pass.
Refactored out _one_audit_pass() method in swift/account/auditor.py and
swift/container/auditor.py.
Fixed some misc. PEP8 violations.
Misc test cleanups and refactorings (particularly the way "fake logging"
is handled).
Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2

Change-Id: I41356ee250c9088a2387b0d493586dd990a04ac3

Please see the doc/source/overview_expiring_objects.rst for
more detail.
Change-Id: I4ab49e731248cf62ce10001016e0c819cc531738