object-expirer: add round_robin_cache_size option

Code Issues Proposed changes

Drive-Bys:
 * DRY out redundent configuration examples in expiring objects overview
 documentation.
 * Add missing delay_reaping man page docs.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I8879dbd13527233c878dff764ec411ce9619ee39

This commit is contained in:

Clay Gerrard

2024年10月15日 10:00:53 -05:00

committed by Alistair Coles

parent 31ef443715

commit df22032d79

8 changed files with 56 additions and 81 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

9
doc/manpages/object-expirer.conf.5

View File

@@ -88,6 +88,15 @@ The default is 1.

The default is 1.

.IP \fBlog_statsd_metric_prefix\fR

The default is empty.

.IP \fBdelay_reaping_account/container\fR

Normally, the expirer begins reaping expired objects immediately. You can add

options prefixed with "delay_reaping_" in the form of

"delay_reaping_<ACCT>[/<CNTR>]" to cause the expirer to delay processing of

tasks in those account or account/container. The <ACCT>[/<CNTR>] part of the

config option names should url-quote the paths. The value is in seconds. The

default is no delay for any tasks.

.IP \fBround_robin_task_cache_size\fR

Number of tasks objects to cache before processing.

.IP \fBnice_priority\fR

Modify scheduling priority of server processes. Niceness values range from -20

(most favorable to the process) to 19 (least favorable to the process).

9
doc/manpages/object-server.conf.5

View File

@@ -623,6 +623,15 @@ to use 3 processes, you should run processes with process set to 0, 1, and 2. Th

The expirer will re-attempt expiring if the source object is not available up

to reclaim_age seconds before it gives up and deletes the task in the queue.

The default is 604800 seconds (= 1 week).

.IP \fBdelay_reaping_account/container\fR

Normally, the expirer begins reaping expired objects immediately. You can add

options prefixed with "delay_reaping_" in the form of

"delay_reaping_<ACCT>[/<CNTR>]" to cause the expirer to delay processing of

tasks in those account or account/container. The <ACCT>[/<CNTR>] part of the

config option names should url-quote the paths. The value is in seconds. The

default is no delay for any tasks.

.IP \fBround_robin_task_cache_size\fR

Number of tasks objects to cache before processing.

.IP \fBrecon_cache_path\fR

Path to recon cache directory. The default is /var/cache/swift

.IP \fBnice_priority\fR

1
doc/source/config/object_server_config.rst

View File

@@ -705,6 +705,7 @@ concurrency 1 Level of concurren

this value must be set to at least 1

expiring_objects_account_name expiring_objects name for legacy expirer task queue

dequeue_from_legacy False This service will look for jobs on the legacy expirer task queue.

round_robin_task_cache_size 100000 Number of tasks objects to cache before processing.

processes 0 How many parts to divide the legacy work into,

one part per process that will be doing the work.

When set 0 means that a single legacy

82
doc/source/overview_expiring_objects.rst

View File

@@ -136,6 +136,11 @@ Upgrading impact: General Task Queue vs Legacy Queue

The expirer daemon will be moving to a new general task-queue based design that

will divide the work across all object servers, as such only expirers defined

in the object-server config will be able to use the new system.

The legacy object expirer config is documented in

``etc/object-expirer.conf-sample``. The alternative object-server config

section is documented in ``etc/object-server.conf-sample``.

The parameters in both files are identical except for a new option in the

object-server ``[object-expirer]`` section, ``dequeue_from_legacy``

which when set to ``True`` will tell the expirer that in addition to using

@@ -176,83 +181,6 @@ the concurrency level for the legacy queue.

containers. On a large cluster one may inadvertently overload the

acccount/container servers handling the legacy expirer queue.

Here is a quick sample of the ``object-expirer`` section required in the

``object-server.conf``::

[object-expirer]

# log_name = object-expirer

# log_facility = LOG_LOCAL0

# log_level = INFO

# log_address = /dev/log

#

interval = 300

# If this true, expirer execute tasks in legacy expirer task queue

dequeue_from_legacy = false

# processes can only be used in conjunction with `dequeue_from_legacy`.

# So this option is ignored if dequeue_from_legacy=false.

# processes is how many parts to divide the legacy work into, one part per

# process that will be doing the work

# processes set 0 means that a single legacy process will be doing all the work

# processes can also be specified on the command line and will override the

# config value

# processes = 0

# process can only be used in conjunction with `dequeue_from_legacy`.

# So this option is ignored if dequeue_from_legacy=false.

# process is which of the parts a particular legacy process will work on

# process can also be specified on the command line and will override the config

# value

# process is "zero based", if you want to use 3 processes, you should run

# processes with process set to 0, 1, and 2

# process = 0

report_interval = 300

# request_tries is the number of times the expirer's internal client will

# attempt any given request in the event of failure. The default is 3.

# request_tries = 3

# concurrency is the level of concurrency to use to do the work, this value

# must be set to at least 1

# concurrency = 1

# The expirer will re-attempt expiring if the source object is not available

# up to reclaim_age seconds before it gives up and deletes the entry in the

# queue.

# reclaim_age = 604800

And for completeness, here is a quick sample of the legacy

``object-expirer.conf`` file::

[DEFAULT]

# swift_dir = /etc/swift

# user = swift

# You can specify default log routing here if you want:

# log_name = swift

# log_facility = LOG_LOCAL0

# log_level = INFO

[object-expirer]

interval = 300

[pipeline:main]

pipeline = catch_errors cache proxy-server

[app:proxy-server]

use = egg:swift#proxy

# See proxy-server.conf-sample for options

[filter:cache]

use = egg:swift#memcache

# See proxy-server.conf-sample for options

[filter:catch_errors]

use = egg:swift#catch_errors

# See proxy-server.conf-sample for options

.. note::

When running legacy expirers, the daemon needs to run on a machine with

access to all the backend servers in the cluster, but does not need proxy