swift

Refactor SWIFT_HASH_PATH_SUFFIX to be in a config file

2010年10月14日 15:58:44 -07:00

[swift-hash]

swift constraints are now settable via config Change previously hard-coded constants into config variables. This allows deployers to tune their cluster more specifically based on their needs. For example, a deployment that uses direct swift access for public content may need to set a larger header value constraint to allow for the full object name to be represented in the Content- Disposition header (which browsers check to determine the name of a downloaded object). The new settings are set in the [swift-constraints] section of /etc/swift/swift.conf. Comments were also added to this config file. Cleaned up swift/common/constraints.py to pass pep8 1.3.3 Funtional tests now require constraints to be defined in /etc/test.conf or in /etc/swift/swift.conf (in the case of running the functional tests against a local swift cluster). To have any hope of tests passing, the defined constraints must match the constraints on the tested cluster. Removed a ton of "magic numbers" in both unit and functional tests. Change-Id: Ie4588e052fd158314ddca6cd8fca9bc793311465

2012年09月05日 20:49:50 -07:00

Fixing /etc/swift.conf-sample to include swift_hash_path_prefix Change-Id: I60f5f3d4083937a03ecb7ed531185c617ea08016

2013年04月22日 19:32:18 +03:00

# swift_hash_path_suffix and swift_hash_path_prefix are used as part of the

Fix typos in Swift files Change-Id: I39dbf55c094c42347b57ef67520abff9e6fc24bc

2016年04月07日 13:31:26 +07:00

# hashing algorithm when determining data placement in the cluster.

Add max_header_size to swift.conf-sample and relative UT 1. Add explanation of MAX_HEADER_SIZE into swift.conf-sample as same as other settings in swift.conf. Especially point out the default size of header line in eventlet is 8192 which is the main reason why we set 8192 for MAX_HEADER_SIZE in swift. 2. Add some unit tests to check valid settings in swift.conf. Test cases in test_constraints use /etc/swift/swift.conf if exists, and if any wrong settings are in it (MAX_META_VALE > MAX_META_OVERALL_SIZE), swift's unit test must fail. These new unit tests is used in this case. Change-Id: I7bb21951d46050163c1b7bceac8d49302b9209f7

2013年06月19日 01:02:37 +08:00

# These values should remain secret and MUST NOT change

# once a cluster has been deployed.

Add note on using printable chars for swift_hash_path_suffix/prefix Using unprintable characters for swift_hash_path_prefix/suffix might lead to hard problems when parsing of these values changes, for example due to newer Python versions or changes in the parsers itself. Let's avoid this and add a note that deployers should use printable strings for these values. Change-Id: I976982b753b6af831ab91d7190f50f8f15bf73bf

2016年02月17日 20:47:26 +01:00

# Use only printable chars (python -c "import string; print(string.printable)")

swift constraints are now settable via config Change previously hard-coded constants into config variables. This allows deployers to tune their cluster more specifically based on their needs. For example, a deployment that uses direct swift access for public content may need to set a larger header value constraint to allow for the full object name to be represented in the Content- Disposition header (which browsers check to determine the name of a downloaded object). The new settings are set in the [swift-constraints] section of /etc/swift/swift.conf. Comments were also added to this config file. Cleaned up swift/common/constraints.py to pass pep8 1.3.3 Funtional tests now require constraints to be defined in /etc/test.conf or in /etc/swift/swift.conf (in the case of running the functional tests against a local swift cluster). To have any hope of tests passing, the defined constraints must match the constraints on the tested cluster. Removed a ton of "magic numbers" in both unit and functional tests. Change-Id: Ie4588e052fd158314ddca6cd8fca9bc793311465

2012年09月05日 20:49:50 -07:00

Refactor SWIFT_HASH_PATH_SUFFIX to be in a config file

2010年10月14日 15:58:44 -07:00

swift_hash_path_suffix = changeme

Fixing /etc/swift.conf-sample to include swift_hash_path_prefix Change-Id: I60f5f3d4083937a03ecb7ed531185c617ea08016

2013年04月22日 19:32:18 +03:00

swift_hash_path_prefix = changeme

swift constraints are now settable via config Change previously hard-coded constants into config variables. This allows deployers to tune their cluster more specifically based on their needs. For example, a deployment that uses direct swift access for public content may need to set a larger header value constraint to allow for the full object name to be represented in the Content- Disposition header (which browsers check to determine the name of a downloaded object). The new settings are set in the [swift-constraints] section of /etc/swift/swift.conf. Comments were also added to this config file. Cleaned up swift/common/constraints.py to pass pep8 1.3.3 Funtional tests now require constraints to be defined in /etc/test.conf or in /etc/swift/swift.conf (in the case of running the functional tests against a local swift cluster). To have any hope of tests passing, the defined constraints must match the constraints on the tested cluster. Removed a ton of "magic numbers" in both unit and functional tests. Change-Id: Ie4588e052fd158314ddca6cd8fca9bc793311465

2012年09月05日 20:49:50 -07:00

Update docs to discourage policy names being numbers There are times when it is convenient to specify a policy by name or by index (see Related-Change), but policy names can unfortunately collide with indexes. Using a number as a policy name should at least be discouraged. Change-Id: I0cdd3b86b527d6656b7fb50c699e3c0cc566e732 Related-Change: Icf1517bd930c74e9552b88250a7b4019e0ab413e

2021年03月25日 10:20:27 +00:00

# Storage policies are defined here and determine various characteristics

# about how objects are stored and treated. More documentation can be found at

# https://docs.openstack.org/swift/latest/overview_policies.html.

# Client requests specify a policy on a per container basis using the policy

# name. Internally the policy name is mapped to the policy index specified in

# the policy's section header in this config file. Policy names are

# case-insensitive and, to avoid confusion with indexes names, should not be

# numbers.

#

# The policy with index 0 is always used for legacy containers and can be given

# a name for use in metadata however the ring file name will always be

# 'object.ring.gz' for backwards compatibility. If no policies are defined a

# policy with index 0 will be automatically created for backwards compatibility

# and given the name Policy-0. A default policy is used when creating new

# containers when no policy is specified in the request. If no other policies

# are defined the policy with index 0 will be declared the default. If

# multiple policies are defined you must define a policy with index 0 and you

# must specify a default. It is recommended you always define a section for

# storage-policy:0.

Add support for policy types, 'erasure_coding' policy This patch extends the StoragePolicy class for non-replication storage policies, the first one being "erasure coding". Changes: - Add 'policy_type' support to BaseStoragePolicy class - Disallow direct instantiation of BaseStoragePolicy class - Subclass BaseStoragePolicy - "StoragePolicy": . Replication policy, default . policy_type = 'replication' - "ECStoragePolicy": . Erasure Coding policy . policy_type = 'erasure_coding' . Private member variables ec_type (EC backend), ec_num_data_fragments (number of fragments original data split into after erasure coding operation), ec_num_parity_fragments (number of parity fragments generated during erasure coding) . Private methods EC specific attributes and ring validator methods. - Swift will use PyECLib, a Python Erasure Coding library, for erasure coding operations. PyECLib is already an approved OpenStack core requirement. (https://bitbucket.org/kmgreen2/pyeclib/) - Add test cases for - 'policy_type' StoragePolicy member - policy_type == 'erasure_coding' DocImpact Co-Authored-By: Alistair Coles <alistair.coles@hp.com> Co-Authored-By: Thiago da Silva <thiago@redhat.com> Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Co-Authored-By: Paul Luse <paul.e.luse@intel.com> Co-Authored-By: Samuel Merritt <sam@swiftstack.com> Co-Authored-By: Christian Schwede <christian.schwede@enovance.com> Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com> Change-Id: Ie0e09796e3ec45d3e656fb7540d0e5a5709b8386 Implements: blueprint ec-proxy-work

2014年06月30日 11:14:28 -07:00

#

# A 'policy_type' argument is also supported but is not mandatory. Default

# policy type 'replication' is used when 'policy_type' is unspecified.

Update docs to discourage policy names being numbers There are times when it is convenient to specify a policy by name or by index (see Related-Change), but policy names can unfortunately collide with indexes. Using a number as a policy name should at least be discouraged. Change-Id: I0cdd3b86b527d6656b7fb50c699e3c0cc566e732 Related-Change: Icf1517bd930c74e9552b88250a7b4019e0ab413e

2021年03月25日 10:20:27 +00:00

#

Configure diskfile per storage policy With this commit, each storage policy can define the diskfile to use to access objects. Selection of the diskfile is done in swift.conf. Example: [storage-policy:0] name = gold policy_type = replication default = yes diskfile = egg:swift#replication.fs The diskfile configuration item accepts the same format than middlewares declaration: [[scheme:]egg_name#]entry_point The egg_name is optional and default to "swift". The scheme is optional and default to the only valid value "egg". The upstream entry points are "replication.fs" and "erasure_coding.fs". Co-Authored-By: Alexandre Lécuyer <alexandre.lecuyer@corp.ovh.com> Co-Authored-By: Alistair Coles <alistairncoles@gmail.com> Change-Id: I070c21bc1eaf1c71ac0652cec9e813cadcc14851

2018年06月29日 16:13:05 +02:00

# A 'diskfile_module' optional argument lets you specify an alternate backend

# object storage plug-in architecture. The default is

# "egg:swift#replication.fs", or "egg:swift#erasure_coding.fs", depending on

# the policy type.

Update docs to discourage policy names being numbers There are times when it is convenient to specify a policy by name or by index (see Related-Change), but policy names can unfortunately collide with indexes. Using a number as a policy name should at least be discouraged. Change-Id: I0cdd3b86b527d6656b7fb50c699e3c0cc566e732 Related-Change: Icf1517bd930c74e9552b88250a7b4019e0ab413e

2021年03月25日 10:20:27 +00:00

#

# Aliases for the storage policy name may be defined, but are not required.

#

Add Storage Policy Support The basic idea here is to replace the use of a single object ring in the Application class with a collection of object rings. The collection includes not only the Ring object itself but the policy name associated with it, the filename for the .gz and any other metadata associated with the policy that may be needed. When containers are created, a policy (thus a specific obj ring) is selected allowing apps to specify policy at container creation time and leverage policies simply by using different containers for object operations. The policy collection is based off of info in the swift.conf file. The format of the sections in the .conf file is as follows: swift.conf format: [storage-policy:0] name = chicken [storage-policy:1] name = turkey default = yes With the above format: - Policy 0 will always be used for access to existing containers without the policy specified. The ring name for policy 0 is always 'object', assuring backwards compatiblity. The parser will always create a policy 0 even if not specified - The policy with 'default=yes' is the one used for new container creation. This allows the admin to specify which policy is used without forcing the application to add the metadata. This commit simply introduces storage policies and the loading thereof; nobody's using it yet. That will follow in subsequent commits. Expose storage policies in /info DocImpact Implements: blueprint storage-policies Change-Id: Ica05f41ecf3adb3648cc9182f11f1c8c5c678985

2014年03月17日 12:18:25 -07:00

[storage-policy:0]

name = Policy-0

default = yes

Add support for policy types, 'erasure_coding' policy This patch extends the StoragePolicy class for non-replication storage policies, the first one being "erasure coding". Changes: - Add 'policy_type' support to BaseStoragePolicy class - Disallow direct instantiation of BaseStoragePolicy class - Subclass BaseStoragePolicy - "StoragePolicy": . Replication policy, default . policy_type = 'replication' - "ECStoragePolicy": . Erasure Coding policy . policy_type = 'erasure_coding' . Private member variables ec_type (EC backend), ec_num_data_fragments (number of fragments original data split into after erasure coding operation), ec_num_parity_fragments (number of parity fragments generated during erasure coding) . Private methods EC specific attributes and ring validator methods. - Swift will use PyECLib, a Python Erasure Coding library, for erasure coding operations. PyECLib is already an approved OpenStack core requirement. (https://bitbucket.org/kmgreen2/pyeclib/) - Add test cases for - 'policy_type' StoragePolicy member - policy_type == 'erasure_coding' DocImpact Co-Authored-By: Alistair Coles <alistair.coles@hp.com> Co-Authored-By: Thiago da Silva <thiago@redhat.com> Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Co-Authored-By: Paul Luse <paul.e.luse@intel.com> Co-Authored-By: Samuel Merritt <sam@swiftstack.com> Co-Authored-By: Christian Schwede <christian.schwede@enovance.com> Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com> Change-Id: Ie0e09796e3ec45d3e656fb7540d0e5a5709b8386 Implements: blueprint ec-proxy-work

2014年06月30日 11:14:28 -07:00

#policy_type = replication

Configure diskfile per storage policy With this commit, each storage policy can define the diskfile to use to access objects. Selection of the diskfile is done in swift.conf. Example: [storage-policy:0] name = gold policy_type = replication default = yes diskfile = egg:swift#replication.fs The diskfile configuration item accepts the same format than middlewares declaration: [[scheme:]egg_name#]entry_point The egg_name is optional and default to "swift". The scheme is optional and default to the only valid value "egg". The upstream entry points are "replication.fs" and "erasure_coding.fs". Co-Authored-By: Alexandre Lécuyer <alexandre.lecuyer@corp.ovh.com> Co-Authored-By: Alistair Coles <alistairncoles@gmail.com> Change-Id: I070c21bc1eaf1c71ac0652cec9e813cadcc14851

2018年06月29日 16:13:05 +02:00

#diskfile_module = egg:swift#replication.fs

Add support for storage policies to have more than one name This patch alters storage_policy.py to allow storage policies to have multiple names. Now users are able to add a number of human-readable aliases for storage policies. Policies now have a .name (the default name), .aliases (a string of comma seperated aliases), and .aliases_list (a list of all human readable names). Policies will always have an .aliases value if no aliases are set it will contain the default name. The policy docs and tests have been updated to reflect changes and policy.get_policy_info has been altered to display the name and aliases Change-Id: I02967ca8d7c790595e5ee551581196aa64552eea

2015年11月05日 23:04:14 +13:00

aliases = yellow, orange

Add Storage Policy Support The basic idea here is to replace the use of a single object ring in the Application class with a collection of object rings. The collection includes not only the Ring object itself but the policy name associated with it, the filename for the .gz and any other metadata associated with the policy that may be needed. When containers are created, a policy (thus a specific obj ring) is selected allowing apps to specify policy at container creation time and leverage policies simply by using different containers for object operations. The policy collection is based off of info in the swift.conf file. The format of the sections in the .conf file is as follows: swift.conf format: [storage-policy:0] name = chicken [storage-policy:1] name = turkey default = yes With the above format: - Policy 0 will always be used for access to existing containers without the policy specified. The ring name for policy 0 is always 'object', assuring backwards compatiblity. The parser will always create a policy 0 even if not specified - The policy with 'default=yes' is the one used for new container creation. This allows the admin to specify which policy is used without forcing the application to add the metadata. This commit simply introduces storage policies and the loading thereof; nobody's using it yet. That will follow in subsequent commits. Expose storage policies in /info DocImpact Implements: blueprint storage-policies Change-Id: Ica05f41ecf3adb3648cc9182f11f1c8c5c678985

2014年03月17日 12:18:25 -07:00

Update docs to discourage policy names being numbers There are times when it is convenient to specify a policy by name or by index (see Related-Change), but policy names can unfortunately collide with indexes. Using a number as a policy name should at least be discouraged. Change-Id: I0cdd3b86b527d6656b7fb50c699e3c0cc566e732 Related-Change: Icf1517bd930c74e9552b88250a7b4019e0ab413e

2021年03月25日 10:20:27 +00:00

# The following section would declare a policy called 'silver', the number of

Add Storage Policy Support The basic idea here is to replace the use of a single object ring in the Application class with a collection of object rings. The collection includes not only the Ring object itself but the policy name associated with it, the filename for the .gz and any other metadata associated with the policy that may be needed. When containers are created, a policy (thus a specific obj ring) is selected allowing apps to specify policy at container creation time and leverage policies simply by using different containers for object operations. The policy collection is based off of info in the swift.conf file. The format of the sections in the .conf file is as follows: swift.conf format: [storage-policy:0] name = chicken [storage-policy:1] name = turkey default = yes With the above format: - Policy 0 will always be used for access to existing containers without the policy specified. The ring name for policy 0 is always 'object', assuring backwards compatiblity. The parser will always create a policy 0 even if not specified - The policy with 'default=yes' is the one used for new container creation. This allows the admin to specify which policy is used without forcing the application to add the metadata. This commit simply introduces storage policies and the loading thereof; nobody's using it yet. That will follow in subsequent commits. Expose storage policies in /info DocImpact Implements: blueprint storage-policies Change-Id: Ica05f41ecf3adb3648cc9182f11f1c8c5c678985

2014年03月17日 12:18:25 -07:00

# replicas will be determined by how the ring is built. In this example the

# 'silver' policy could have a lower or higher # of replicas than the

# 'Policy-0' policy above. The ring filename will be 'object-1.ring.gz'. You

# may only specify one storage policy section as the default. If you changed

# this section to specify 'silver' as the default, when a client created a new

# container w/o a policy specified, it will get the 'silver' policy because

# this config has specified it as the default. However if a legacy container

# (one created with a pre-policy version of swift) is accessed, it is known

# implicitly to be assigned to the policy with index 0 as opposed to the

Add support for storage policies to have more than one name This patch alters storage_policy.py to allow storage policies to have multiple names. Now users are able to add a number of human-readable aliases for storage policies. Policies now have a .name (the default name), .aliases (a string of comma seperated aliases), and .aliases_list (a list of all human readable names). Policies will always have an .aliases value if no aliases are set it will contain the default name. The policy docs and tests have been updated to reflect changes and policy.get_policy_info has been altered to display the name and aliases Change-Id: I02967ca8d7c790595e5ee551581196aa64552eea

2015年11月05日 23:04:14 +13:00

# current default. Note that even without specifying any aliases, a policy

# always has at least the default name stored in aliases because this field is

# used to contain all human readable names for a storage policy.

#

Add Storage Policy Support The basic idea here is to replace the use of a single object ring in the Application class with a collection of object rings. The collection includes not only the Ring object itself but the policy name associated with it, the filename for the .gz and any other metadata associated with the policy that may be needed. When containers are created, a policy (thus a specific obj ring) is selected allowing apps to specify policy at container creation time and leverage policies simply by using different containers for object operations. The policy collection is based off of info in the swift.conf file. The format of the sections in the .conf file is as follows: swift.conf format: [storage-policy:0] name = chicken [storage-policy:1] name = turkey default = yes With the above format: - Policy 0 will always be used for access to existing containers without the policy specified. The ring name for policy 0 is always 'object', assuring backwards compatiblity. The parser will always create a policy 0 even if not specified - The policy with 'default=yes' is the one used for new container creation. This allows the admin to specify which policy is used without forcing the application to add the metadata. This commit simply introduces storage policies and the loading thereof; nobody's using it yet. That will follow in subsequent commits. Expose storage policies in /info DocImpact Implements: blueprint storage-policies Change-Id: Ica05f41ecf3adb3648cc9182f11f1c8c5c678985

2014年03月17日 12:18:25 -07:00

#[storage-policy:1]

#name = silver

Add support for policy types, 'erasure_coding' policy This patch extends the StoragePolicy class for non-replication storage policies, the first one being "erasure coding". Changes: - Add 'policy_type' support to BaseStoragePolicy class - Disallow direct instantiation of BaseStoragePolicy class - Subclass BaseStoragePolicy - "StoragePolicy": . Replication policy, default . policy_type = 'replication' - "ECStoragePolicy": . Erasure Coding policy . policy_type = 'erasure_coding' . Private member variables ec_type (EC backend), ec_num_data_fragments (number of fragments original data split into after erasure coding operation), ec_num_parity_fragments (number of parity fragments generated during erasure coding) . Private methods EC specific attributes and ring validator methods. - Swift will use PyECLib, a Python Erasure Coding library, for erasure coding operations. PyECLib is already an approved OpenStack core requirement. (https://bitbucket.org/kmgreen2/pyeclib/) - Add test cases for - 'policy_type' StoragePolicy member - policy_type == 'erasure_coding' DocImpact Co-Authored-By: Alistair Coles <alistair.coles@hp.com> Co-Authored-By: Thiago da Silva <thiago@redhat.com> Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Co-Authored-By: Paul Luse <paul.e.luse@intel.com> Co-Authored-By: Samuel Merritt <sam@swiftstack.com> Co-Authored-By: Christian Schwede <christian.schwede@enovance.com> Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com> Change-Id: Ie0e09796e3ec45d3e656fb7540d0e5a5709b8386 Implements: blueprint ec-proxy-work

2014年06月30日 11:14:28 -07:00

#policy_type = replication

Configure diskfile per storage policy With this commit, each storage policy can define the diskfile to use to access objects. Selection of the diskfile is done in swift.conf. Example: [storage-policy:0] name = gold policy_type = replication default = yes diskfile = egg:swift#replication.fs The diskfile configuration item accepts the same format than middlewares declaration: [[scheme:]egg_name#]entry_point The egg_name is optional and default to "swift". The scheme is optional and default to the only valid value "egg". The upstream entry points are "replication.fs" and "erasure_coding.fs". Co-Authored-By: Alexandre Lécuyer <alexandre.lecuyer@corp.ovh.com> Co-Authored-By: Alistair Coles <alistairncoles@gmail.com> Change-Id: I070c21bc1eaf1c71ac0652cec9e813cadcc14851

2018年06月29日 16:13:05 +02:00

#diskfile_module = egg:swift#replication.fs

Add support for policy types, 'erasure_coding' policy This patch extends the StoragePolicy class for non-replication storage policies, the first one being "erasure coding". Changes: - Add 'policy_type' support to BaseStoragePolicy class - Disallow direct instantiation of BaseStoragePolicy class - Subclass BaseStoragePolicy - "StoragePolicy": . Replication policy, default . policy_type = 'replication' - "ECStoragePolicy": . Erasure Coding policy . policy_type = 'erasure_coding' . Private member variables ec_type (EC backend), ec_num_data_fragments (number of fragments original data split into after erasure coding operation), ec_num_parity_fragments (number of parity fragments generated during erasure coding) . Private methods EC specific attributes and ring validator methods. - Swift will use PyECLib, a Python Erasure Coding library, for erasure coding operations. PyECLib is already an approved OpenStack core requirement. (https://bitbucket.org/kmgreen2/pyeclib/) - Add test cases for - 'policy_type' StoragePolicy member - policy_type == 'erasure_coding' DocImpact Co-Authored-By: Alistair Coles <alistair.coles@hp.com> Co-Authored-By: Thiago da Silva <thiago@redhat.com> Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Co-Authored-By: Paul Luse <paul.e.luse@intel.com> Co-Authored-By: Samuel Merritt <sam@swiftstack.com> Co-Authored-By: Christian Schwede <christian.schwede@enovance.com> Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com> Change-Id: Ie0e09796e3ec45d3e656fb7540d0e5a5709b8386 Implements: blueprint ec-proxy-work

2014年06月30日 11:14:28 -07:00

# The following declares a storage policy of type 'erasure_coding' which uses

Remove Erasure Coding beta status from docs This removes notes stating support for Erasure coding as beta. Questions regarding the stability of EC are coming up regularly, and are often referring to the docs that state EC as still in beta. Besides this, a note marking statsd support as beta has been removed as well. Change-Id: If4fb6a5c4cb741d42953db3cee8cb17a1d774e15

2016年03月04日 09:33:17 +00:00

# Erasure Coding for data reliability. Please refer to Swift documentation for

Add support for policy types, 'erasure_coding' policy This patch extends the StoragePolicy class for non-replication storage policies, the first one being "erasure coding". Changes: - Add 'policy_type' support to BaseStoragePolicy class - Disallow direct instantiation of BaseStoragePolicy class - Subclass BaseStoragePolicy - "StoragePolicy": . Replication policy, default . policy_type = 'replication' - "ECStoragePolicy": . Erasure Coding policy . policy_type = 'erasure_coding' . Private member variables ec_type (EC backend), ec_num_data_fragments (number of fragments original data split into after erasure coding operation), ec_num_parity_fragments (number of parity fragments generated during erasure coding) . Private methods EC specific attributes and ring validator methods. - Swift will use PyECLib, a Python Erasure Coding library, for erasure coding operations. PyECLib is already an approved OpenStack core requirement. (https://bitbucket.org/kmgreen2/pyeclib/) - Add test cases for - 'policy_type' StoragePolicy member - policy_type == 'erasure_coding' DocImpact Co-Authored-By: Alistair Coles <alistair.coles@hp.com> Co-Authored-By: Thiago da Silva <thiago@redhat.com> Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Co-Authored-By: Paul Luse <paul.e.luse@intel.com> Co-Authored-By: Samuel Merritt <sam@swiftstack.com> Co-Authored-By: Christian Schwede <christian.schwede@enovance.com> Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com> Change-Id: Ie0e09796e3ec45d3e656fb7540d0e5a5709b8386 Implements: blueprint ec-proxy-work

2014年06月30日 11:14:28 -07:00

# details on how the 'erasure_coding' storage policy is implemented.

#

# Swift uses PyECLib, a Python Erasure coding API library, for encode/decode

# operations. Please refer to Swift documentation for details on how to

# install PyECLib.

#

# When defining an EC policy, 'policy_type' needs to be 'erasure_coding' and

# EC configuration parameters 'ec_type', 'ec_num_data_fragments' and

# 'ec_num_parity_fragments' must be specified. 'ec_type' is chosen from the

# list of EC backends supported by PyECLib. The ring configured for the

Require that known-bad EC schemes be deprecated We said we were going to do it, we've had two releases saying we'd do it, we've even backported our saying it to Newton -- let's actually do it. Upgrade Consideration ===================== Erasure-coded storage policies using isa_l_rs_vand and nparity >= 5 must be configured as deprecated, preventing any new containers from being created with such a policy. This configuration is known to harm data durability. Any data in such policies should be migrated to a new policy. See https://bugs.launchpad.net/swift/+bug/1639691 for more information. UpgradeImpact Related-Change: I50159c9d19f2385d5f60112e9aaefa1a68098313 Change-Id: I8f9de0bec01032d9d9b58848e2a76ac92e65ab09 Closes-Bug: 1639691

2017年05月25日 09:40:47 -07:00

# storage policy must have its "replica" count configured to

Add support for policy types, 'erasure_coding' policy This patch extends the StoragePolicy class for non-replication storage policies, the first one being "erasure coding". Changes: - Add 'policy_type' support to BaseStoragePolicy class - Disallow direct instantiation of BaseStoragePolicy class - Subclass BaseStoragePolicy - "StoragePolicy": . Replication policy, default . policy_type = 'replication' - "ECStoragePolicy": . Erasure Coding policy . policy_type = 'erasure_coding' . Private member variables ec_type (EC backend), ec_num_data_fragments (number of fragments original data split into after erasure coding operation), ec_num_parity_fragments (number of parity fragments generated during erasure coding) . Private methods EC specific attributes and ring validator methods. - Swift will use PyECLib, a Python Erasure Coding library, for erasure coding operations. PyECLib is already an approved OpenStack core requirement. (https://bitbucket.org/kmgreen2/pyeclib/) - Add test cases for - 'policy_type' StoragePolicy member - policy_type == 'erasure_coding' DocImpact Co-Authored-By: Alistair Coles <alistair.coles@hp.com> Co-Authored-By: Thiago da Silva <thiago@redhat.com> Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Co-Authored-By: Paul Luse <paul.e.luse@intel.com> Co-Authored-By: Samuel Merritt <sam@swiftstack.com> Co-Authored-By: Christian Schwede <christian.schwede@enovance.com> Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com> Change-Id: Ie0e09796e3ec45d3e656fb7540d0e5a5709b8386 Implements: blueprint ec-proxy-work

2014年06月30日 11:14:28 -07:00

# 'ec_num_data_fragments' + 'ec_num_parity_fragments' - this requirement is

# validated when services start. 'ec_object_segment_size' is the amount of

# data that will be buffered up before feeding a segment into the

# encoder/decoder. More information about these configuration options and

Require that known-bad EC schemes be deprecated We said we were going to do it, we've had two releases saying we'd do it, we've even backported our saying it to Newton -- let's actually do it. Upgrade Consideration ===================== Erasure-coded storage policies using isa_l_rs_vand and nparity >= 5 must be configured as deprecated, preventing any new containers from being created with such a policy. This configuration is known to harm data durability. Any data in such policies should be migrated to a new policy. See https://bugs.launchpad.net/swift/+bug/1639691 for more information. UpgradeImpact Related-Change: I50159c9d19f2385d5f60112e9aaefa1a68098313 Change-Id: I8f9de0bec01032d9d9b58848e2a76ac92e65ab09 Closes-Bug: 1639691

2017年05月25日 09:40:47 -07:00

# supported 'ec_type' schemes is available in the Swift documentation. See

Update and optimize documentation links * Update URLs according to document migration * Update the dead and outdated links Change-Id: Id92552f4a2d0fb79ddefc55a08636f2e7aeb07cb

2017年07月20日 15:09:52 +08:00

# https://docs.openstack.org/swift/latest/overview_erasure_code.html

Require that known-bad EC schemes be deprecated We said we were going to do it, we've had two releases saying we'd do it, we've even backported our saying it to Newton -- let's actually do it. Upgrade Consideration ===================== Erasure-coded storage policies using isa_l_rs_vand and nparity >= 5 must be configured as deprecated, preventing any new containers from being created with such a policy. This configuration is known to harm data durability. Any data in such policies should be migrated to a new policy. See https://bugs.launchpad.net/swift/+bug/1639691 for more information. UpgradeImpact Related-Change: I50159c9d19f2385d5f60112e9aaefa1a68098313 Change-Id: I8f9de0bec01032d9d9b58848e2a76ac92e65ab09 Closes-Bug: 1639691

2017年05月25日 09:40:47 -07:00

# for more information on how to configure EC policies.

Add support for policy types, 'erasure_coding' policy This patch extends the StoragePolicy class for non-replication storage policies, the first one being "erasure coding". Changes: - Add 'policy_type' support to BaseStoragePolicy class - Disallow direct instantiation of BaseStoragePolicy class - Subclass BaseStoragePolicy - "StoragePolicy": . Replication policy, default . policy_type = 'replication' - "ECStoragePolicy": . Erasure Coding policy . policy_type = 'erasure_coding' . Private member variables ec_type (EC backend), ec_num_data_fragments (number of fragments original data split into after erasure coding operation), ec_num_parity_fragments (number of parity fragments generated during erasure coding) . Private methods EC specific attributes and ring validator methods. - Swift will use PyECLib, a Python Erasure Coding library, for erasure coding operations. PyECLib is already an approved OpenStack core requirement. (https://bitbucket.org/kmgreen2/pyeclib/) - Add test cases for - 'policy_type' StoragePolicy member - policy_type == 'erasure_coding' DocImpact Co-Authored-By: Alistair Coles <alistair.coles@hp.com> Co-Authored-By: Thiago da Silva <thiago@redhat.com> Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Co-Authored-By: Paul Luse <paul.e.luse@intel.com> Co-Authored-By: Samuel Merritt <sam@swiftstack.com> Co-Authored-By: Christian Schwede <christian.schwede@enovance.com> Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com> Change-Id: Ie0e09796e3ec45d3e656fb7540d0e5a5709b8386 Implements: blueprint ec-proxy-work

2014年06月30日 11:14:28 -07:00

#

# The example 'deepfreeze10-4' policy defined below is a _sample_

Add support for storage policies to have more than one name This patch alters storage_policy.py to allow storage policies to have multiple names. Now users are able to add a number of human-readable aliases for storage policies. Policies now have a .name (the default name), .aliases (a string of comma seperated aliases), and .aliases_list (a list of all human readable names). Policies will always have an .aliases value if no aliases are set it will contain the default name. The policy docs and tests have been updated to reflect changes and policy.get_policy_info has been altered to display the name and aliases Change-Id: I02967ca8d7c790595e5ee551581196aa64552eea

2015年11月05日 23:04:14 +13:00

# configuration with an alias of 'df10-4' as well as 10 'data' and 4 'parity'

# fragments. 'ec_type' defines the Erasure Coding scheme.

Changed EC backend from jerasure to liberasurecode in examples and docs liberasurecode_rs_vand is build-in liberasurecode, so you don't need another depedency libjerasure2. liberasurecode_rs_vand is supported by pyeclib from 1.0.8 version, so bumping version up. Closes-Bug: #1534325 Change-Id: If2d96875694df8fd48c5278395859aaa165cb566

2016年01月31日 10:30:06 +01:00

# 'liberasurecode_rs_vand' (Reed-Solomon Vandermonde) is used as an example

# below.

Add support for policy types, 'erasure_coding' policy This patch extends the StoragePolicy class for non-replication storage policies, the first one being "erasure coding". Changes: - Add 'policy_type' support to BaseStoragePolicy class - Disallow direct instantiation of BaseStoragePolicy class - Subclass BaseStoragePolicy - "StoragePolicy": . Replication policy, default . policy_type = 'replication' - "ECStoragePolicy": . Erasure Coding policy . policy_type = 'erasure_coding' . Private member variables ec_type (EC backend), ec_num_data_fragments (number of fragments original data split into after erasure coding operation), ec_num_parity_fragments (number of parity fragments generated during erasure coding) . Private methods EC specific attributes and ring validator methods. - Swift will use PyECLib, a Python Erasure Coding library, for erasure coding operations. PyECLib is already an approved OpenStack core requirement. (https://bitbucket.org/kmgreen2/pyeclib/) - Add test cases for - 'policy_type' StoragePolicy member - policy_type == 'erasure_coding' DocImpact Co-Authored-By: Alistair Coles <alistair.coles@hp.com> Co-Authored-By: Thiago da Silva <thiago@redhat.com> Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Co-Authored-By: Paul Luse <paul.e.luse@intel.com> Co-Authored-By: Samuel Merritt <sam@swiftstack.com> Co-Authored-By: Christian Schwede <christian.schwede@enovance.com> Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com> Change-Id: Ie0e09796e3ec45d3e656fb7540d0e5a5709b8386 Implements: blueprint ec-proxy-work

2014年06月30日 11:14:28 -07:00

#

#[storage-policy:2]

#name = deepfreeze10-4

Add support for storage policies to have more than one name This patch alters storage_policy.py to allow storage policies to have multiple names. Now users are able to add a number of human-readable aliases for storage policies. Policies now have a .name (the default name), .aliases (a string of comma seperated aliases), and .aliases_list (a list of all human readable names). Policies will always have an .aliases value if no aliases are set it will contain the default name. The policy docs and tests have been updated to reflect changes and policy.get_policy_info has been altered to display the name and aliases Change-Id: I02967ca8d7c790595e5ee551581196aa64552eea

2015年11月05日 23:04:14 +13:00

#aliases = df10-4

Add support for policy types, 'erasure_coding' policy This patch extends the StoragePolicy class for non-replication storage policies, the first one being "erasure coding". Changes: - Add 'policy_type' support to BaseStoragePolicy class - Disallow direct instantiation of BaseStoragePolicy class - Subclass BaseStoragePolicy - "StoragePolicy": . Replication policy, default . policy_type = 'replication' - "ECStoragePolicy": . Erasure Coding policy . policy_type = 'erasure_coding' . Private member variables ec_type (EC backend), ec_num_data_fragments (number of fragments original data split into after erasure coding operation), ec_num_parity_fragments (number of parity fragments generated during erasure coding) . Private methods EC specific attributes and ring validator methods. - Swift will use PyECLib, a Python Erasure Coding library, for erasure coding operations. PyECLib is already an approved OpenStack core requirement. (https://bitbucket.org/kmgreen2/pyeclib/) - Add test cases for - 'policy_type' StoragePolicy member - policy_type == 'erasure_coding' DocImpact Co-Authored-By: Alistair Coles <alistair.coles@hp.com> Co-Authored-By: Thiago da Silva <thiago@redhat.com> Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Co-Authored-By: Paul Luse <paul.e.luse@intel.com> Co-Authored-By: Samuel Merritt <sam@swiftstack.com> Co-Authored-By: Christian Schwede <christian.schwede@enovance.com> Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com> Change-Id: Ie0e09796e3ec45d3e656fb7540d0e5a5709b8386 Implements: blueprint ec-proxy-work

2014年06月30日 11:14:28 -07:00

#policy_type = erasure_coding

Configure diskfile per storage policy With this commit, each storage policy can define the diskfile to use to access objects. Selection of the diskfile is done in swift.conf. Example: [storage-policy:0] name = gold policy_type = replication default = yes diskfile = egg:swift#replication.fs The diskfile configuration item accepts the same format than middlewares declaration: [[scheme:]egg_name#]entry_point The egg_name is optional and default to "swift". The scheme is optional and default to the only valid value "egg". The upstream entry points are "replication.fs" and "erasure_coding.fs". Co-Authored-By: Alexandre Lécuyer <alexandre.lecuyer@corp.ovh.com> Co-Authored-By: Alistair Coles <alistairncoles@gmail.com> Change-Id: I070c21bc1eaf1c71ac0652cec9e813cadcc14851

2018年06月29日 16:13:05 +02:00

#diskfile_module = egg:swift#erasure_coding.fs

Changed EC backend from jerasure to liberasurecode in examples and docs liberasurecode_rs_vand is build-in liberasurecode, so you don't need another depedency libjerasure2. liberasurecode_rs_vand is supported by pyeclib from 1.0.8 version, so bumping version up. Closes-Bug: #1534325 Change-Id: If2d96875694df8fd48c5278395859aaa165cb566

2016年01月31日 10:30:06 +01:00

#ec_type = liberasurecode_rs_vand

Add support for policy types, 'erasure_coding' policy This patch extends the StoragePolicy class for non-replication storage policies, the first one being "erasure coding". Changes: - Add 'policy_type' support to BaseStoragePolicy class - Disallow direct instantiation of BaseStoragePolicy class - Subclass BaseStoragePolicy - "StoragePolicy": . Replication policy, default . policy_type = 'replication' - "ECStoragePolicy": . Erasure Coding policy . policy_type = 'erasure_coding' . Private member variables ec_type (EC backend), ec_num_data_fragments (number of fragments original data split into after erasure coding operation), ec_num_parity_fragments (number of parity fragments generated during erasure coding) . Private methods EC specific attributes and ring validator methods. - Swift will use PyECLib, a Python Erasure Coding library, for erasure coding operations. PyECLib is already an approved OpenStack core requirement. (https://bitbucket.org/kmgreen2/pyeclib/) - Add test cases for - 'policy_type' StoragePolicy member - policy_type == 'erasure_coding' DocImpact Co-Authored-By: Alistair Coles <alistair.coles@hp.com> Co-Authored-By: Thiago da Silva <thiago@redhat.com> Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Co-Authored-By: Paul Luse <paul.e.luse@intel.com> Co-Authored-By: Samuel Merritt <sam@swiftstack.com> Co-Authored-By: Christian Schwede <christian.schwede@enovance.com> Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com> Change-Id: Ie0e09796e3ec45d3e656fb7540d0e5a5709b8386 Implements: blueprint ec-proxy-work

2014年06月30日 11:14:28 -07:00

#ec_num_data_fragments = 10

#ec_num_parity_fragments = 4

#ec_object_segment_size = 1048576

EC Fragment Duplication - Foundational Global EC Cluster Support This patch enables efficent PUT/GET for global distributed cluster[1]. Problem: Erasure coding has the capability to decrease the amout of actual stored data less then replicated model. For example, ec_k=6, ec_m=3 parameter can be 1.5x of the original data which is smaller than 3x replicated. However, unlike replication, erasure coding requires availability of at least some ec_k fragments of the total ec_k + ec_m fragments to service read (e.g. 6 of 9 in the case above). As such, if we stored the EC object into a swift cluster on 2 geographically distributed data centers which have the same volume of disks, it is likely the fragments will be stored evenly (about 4 and 5) so we still need to access a faraway data center to decode the original object. In addition, if one of the data centers was lost in a disaster, the stored objects will be lost forever, and we have to cry a lot. To ensure highly durable storage, you would think of making *more* parity fragments (e.g. ec_k=6, ec_m=10), unfortunately this causes *significant* performance degradation due to the cost of mathmetical caluculation for erasure coding encode/decode. How this resolves the problem: EC Fragment Duplication extends on the initial solution to add *more* fragments from which to rebuild an object similar to the solution described above. The difference is making *copies* of encoded fragments. With experimental results[1][2], employing small ec_k and ec_m shows enough performance to store/retrieve objects. On PUT: - Encode incomming object with small ec_k and ec_m <- faster! - Make duplicated copies of the encoded fragments. The # of copies are determined by 'ec_duplication_factor' in swift.conf - Store all fragments in Swift Global EC Cluster The duplicated fragments increase pressure on existing requirements when decoding objects in service to a read request. All fragments are stored with their X-Object-Sysmeta-Ec-Frag-Index. In this change, the X-Object-Sysmeta-Ec-Frag-Index represents the actual fragment index encoded by PyECLib, there *will* be duplicates. Anytime we must decode the original object data, we must only consider the ec_k fragments as unique according to their X-Object-Sysmeta-Ec-Frag-Index. On decode no duplicate X-Object-Sysmeta-Ec-Frag-Index may be used when decoding an object, duplicate X-Object-Sysmeta-Ec-Frag-Index should be expected and avoided if possible. On GET: This patch inclues following changes: - Change GET Path to sort primary nodes grouping as subsets, so that each subset will includes unique fragments - Change Reconstructor to be more aware of possibly duplicate fragments For example, with this change, a policy could be configured such that swift.conf: ec_num_data_fragments = 2 ec_num_parity_fragments = 1 ec_duplication_factor = 2 (object ring must have 6 replicas) At Object-Server: node index (from object ring): 0 1 2 3 4 5 <- keep node index for reconstruct decision X-Object-Sysmeta-Ec-Frag-Index: 0 1 2 0 1 2 <- each object keeps actual fragment index for backend (PyEClib) Additional improvements to Global EC Cluster Support will require features such as Composite Rings, and more efficient fragment rebalance/reconstruction. 1: http://goo.gl/IYiNPk (Swift Design Spec Repository) 2: http://goo.gl/frgj6w (Slide Share for OpenStack Summit Tokyo) Doc-Impact Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Change-Id: Idd155401982a2c48110c30b480966a863f6bd305

2015年08月06日 01:06:47 -07:00

#

# Duplicated EC fragments is proof-of-concept experimental support to enable

# Global Erasure Coding policies with multiple regions acting as independent

# failure domains. Do not change the default except in development/testing.

#ec_duplication_factor = 1

swift constraints are now settable via config Change previously hard-coded constants into config variables. This allows deployers to tune their cluster more specifically based on their needs. For example, a deployment that uses direct swift access for public content may need to set a larger header value constraint to allow for the full object name to be represented in the Content- Disposition header (which browsers check to determine the name of a downloaded object). The new settings are set in the [swift-constraints] section of /etc/swift/swift.conf. Comments were also added to this config file. Cleaned up swift/common/constraints.py to pass pep8 1.3.3 Funtional tests now require constraints to be defined in /etc/test.conf or in /etc/swift/swift.conf (in the case of running the functional tests against a local swift cluster). To have any hope of tests passing, the defined constraints must match the constraints on the tested cluster. Removed a ton of "magic numbers" in both unit and functional tests. Change-Id: Ie4588e052fd158314ddca6cd8fca9bc793311465

2012年09月05日 20:49:50 -07:00

# The swift-constraints section sets the basic constraints on data

Add support for policy types, 'erasure_coding' policy This patch extends the StoragePolicy class for non-replication storage policies, the first one being "erasure coding". Changes: - Add 'policy_type' support to BaseStoragePolicy class - Disallow direct instantiation of BaseStoragePolicy class - Subclass BaseStoragePolicy - "StoragePolicy": . Replication policy, default . policy_type = 'replication' - "ECStoragePolicy": . Erasure Coding policy . policy_type = 'erasure_coding' . Private member variables ec_type (EC backend), ec_num_data_fragments (number of fragments original data split into after erasure coding operation), ec_num_parity_fragments (number of parity fragments generated during erasure coding) . Private methods EC specific attributes and ring validator methods. - Swift will use PyECLib, a Python Erasure Coding library, for erasure coding operations. PyECLib is already an approved OpenStack core requirement. (https://bitbucket.org/kmgreen2/pyeclib/) - Add test cases for - 'policy_type' StoragePolicy member - policy_type == 'erasure_coding' DocImpact Co-Authored-By: Alistair Coles <alistair.coles@hp.com> Co-Authored-By: Thiago da Silva <thiago@redhat.com> Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Co-Authored-By: Paul Luse <paul.e.luse@intel.com> Co-Authored-By: Samuel Merritt <sam@swiftstack.com> Co-Authored-By: Christian Schwede <christian.schwede@enovance.com> Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com> Change-Id: Ie0e09796e3ec45d3e656fb7540d0e5a5709b8386 Implements: blueprint ec-proxy-work

2014年06月30日 11:14:28 -07:00

# saved in the swift cluster. These constraints are automatically

Add tests and comments re constraints in /info Add test to check that only the expected keys are reported by proxy in /info, and add comments to raise awareness that default constraints will be automatically published by proxy in response to /info requests. Change-Id: Ia5f6339b06cdc2e1dc960d1f75562a2505530202

2014年04月08日 18:44:06 +01:00

# published by the proxy server in responses to /info requests.

swift constraints are now settable via config Change previously hard-coded constants into config variables. This allows deployers to tune their cluster more specifically based on their needs. For example, a deployment that uses direct swift access for public content may need to set a larger header value constraint to allow for the full object name to be represented in the Content- Disposition header (which browsers check to determine the name of a downloaded object). The new settings are set in the [swift-constraints] section of /etc/swift/swift.conf. Comments were also added to this config file. Cleaned up swift/common/constraints.py to pass pep8 1.3.3 Funtional tests now require constraints to be defined in /etc/test.conf or in /etc/swift/swift.conf (in the case of running the functional tests against a local swift cluster). To have any hope of tests passing, the defined constraints must match the constraints on the tested cluster. Removed a ton of "magic numbers" in both unit and functional tests. Change-Id: Ie4588e052fd158314ddca6cd8fca9bc793311465

2012年09月05日 20:49:50 -07:00

[swift-constraints]

# max_file_size is the largest "normal" object that can be saved in

# the cluster. This is also the limit on the size of each segment of

# a "large" object when using the large object manifest support.

# This value is set in bytes. Setting it to lower than 1MiB will cause

# some tests to fail. It is STRONGLY recommended to leave this value at

# the default (5 * 2**30 + 2).

#max_file_size = 5368709122

# max_meta_name_length is the max number of bytes in the utf8 encoding

# of the name portion of a metadata header.

#max_meta_name_length = 128

# max_meta_value_length is the max number of bytes in the utf8 encoding

# of a metadata value

#max_meta_value_length = 256

# max_meta_count is the max number of metadata keys that can be stored

# on a single account, container, or object

#max_meta_count = 90

# max_meta_overall_size is the max number of bytes in the utf8 encoding

# of the metadata (keys + values)

#max_meta_overall_size = 4096

Add max_header_size to swift.conf-sample and relative UT 1. Add explanation of MAX_HEADER_SIZE into swift.conf-sample as same as other settings in swift.conf. Especially point out the default size of header line in eventlet is 8192 which is the main reason why we set 8192 for MAX_HEADER_SIZE in swift. 2. Add some unit tests to check valid settings in swift.conf. Test cases in test_constraints use /etc/swift/swift.conf if exists, and if any wrong settings are in it (MAX_META_VALE > MAX_META_OVERALL_SIZE), swift's unit test must fail. These new unit tests is used in this case. Change-Id: I7bb21951d46050163c1b7bceac8d49302b9209f7

2013年06月19日 01:02:37 +08:00

# max_header_size is the max number of bytes in the utf8 encoding of each

Fix Error 400 Header Line Too Long Fix Error 400 Header Line Too Long when using Identity v3 PKI Tokens Uses swift.conf max_header_size option to set wsgi.MAX_HEADER_LINE, allowing the operator to customize this parameter. The default value has been let to 8192 to avoid unexpected configuration change on deployed platforms. The max_header_size option has to be increased (for example to 16384), to accomodate for large Identity v3 PKI tokens, including more than 7 catalog entries. The default max header line size of 8192 is exceeded in the following scenario: - Auth tokens generated by Keystone v3 API include the catalog. - Keystone's catalog contains more than 7 services. Similar fixes have been merged in other projects. Change-Id: Ia838b18331f57dfd02b9f71d4523d4059f38e600 Closes-Bug: 1190149

2014年01月14日 15:58:38 +00:00

# header. Using 8192 as default because eventlet use 8192 as max size of

# header line. This value may need to be increased when using identity

# v3 API tokens including more than 7 catalog entries.

# See also include_service_catalog in proxy-server.conf-sample

Update docs to discourage policy names being numbers There are times when it is convenient to specify a policy by name or by index (see Related-Change), but policy names can unfortunately collide with indexes. Using a number as a policy name should at least be discouraged. Change-Id: I0cdd3b86b527d6656b7fb50c699e3c0cc566e732 Related-Change: Icf1517bd930c74e9552b88250a7b4019e0ab413e

2021年03月25日 10:20:27 +00:00

# (documented at https://docs.openstack.org/swift/latest/overview_auth.html)

Add max_header_size to swift.conf-sample and relative UT 1. Add explanation of MAX_HEADER_SIZE into swift.conf-sample as same as other settings in swift.conf. Especially point out the default size of header line in eventlet is 8192 which is the main reason why we set 8192 for MAX_HEADER_SIZE in swift. 2. Add some unit tests to check valid settings in swift.conf. Test cases in test_constraints use /etc/swift/swift.conf if exists, and if any wrong settings are in it (MAX_META_VALE > MAX_META_OVERALL_SIZE), swift's unit test must fail. These new unit tests is used in this case. Change-Id: I7bb21951d46050163c1b7bceac8d49302b9209f7

2013年06月19日 01:02:37 +08:00

#max_header_size = 8192

Increase httplib._MAXHEADERS Python 2.7.9+ and 3.2.6+ limits the number of maximum headers in httplib to 100 [1,2,3]. This setting is too low for Swift. By default the maximum number of allowed headers depends on the number of max allowed metadata settings plus a default value of 32 for regular http headers. If for some reason this is not enough (custom middleware for example) it can be increased with the extra_header_count constraint. [1] https://bugs.python.org/issue16037 [2] https://hg.python.org/cpython/raw-file/15c95b7d81dc/Misc/NEWS [3] https://hg.python.org/cpython/raw-file/v3.2.6/Misc/NEWS Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Co-Authored-By: Matthew Oliver <matt@oliver.net.au> Co-Authored-By: Thomas Herve <therve@redhat.com> Change-Id: I388fd697ec88476024b0e9f1ae75ba35ff765282

2015年06月26日 08:54:22 +00:00

# By default the maximum number of allowed headers depends on the number of max

Enable object body and metadata encryption Adds encryption middlewares. All object servers and proxy servers should be upgraded before introducing encryption middleware. Encryption middleware should be first introduced with the encryption middleware disable_encryption option set to True. Once all proxies have encryption middleware installed this option may be set to False (the default). Increases constraints.py:MAX_HEADER_COUNT by 4 to allow for headers generated by encryption-related middleware. Co-Authored-By: Tim Burke <tim.burke@gmail.com> Co-Authored-By: Christian Cachin <cca@zurich.ibm.com> Co-Authored-By: Mahati Chamarthy <mahati.chamarthy@gmail.com> Co-Authored-By: Peter Chng <pchng@ca.ibm.com> Co-Authored-By: Alistair Coles <alistair.coles@hpe.com> Co-Authored-By: Jonathan Hinson <jlhinson@us.ibm.com> Co-Authored-By: Hamdi Roumani <roumani@ca.ibm.com> UpgradeImpact Change-Id: Ie6db22697ceb1021baaa6bddcf8e41ae3acb5376

2016年06月07日 15:01:32 +01:00

# allowed metadata settings plus a default value of 36 for swift internally

# generated headers and regular http headers. If for some reason this is not

# enough (custom middleware for example) it can be increased with the

# extra_header_count constraint.

Increase httplib._MAXHEADERS Python 2.7.9+ and 3.2.6+ limits the number of maximum headers in httplib to 100 [1,2,3]. This setting is too low for Swift. By default the maximum number of allowed headers depends on the number of max allowed metadata settings plus a default value of 32 for regular http headers. If for some reason this is not enough (custom middleware for example) it can be increased with the extra_header_count constraint. [1] https://bugs.python.org/issue16037 [2] https://hg.python.org/cpython/raw-file/15c95b7d81dc/Misc/NEWS [3] https://hg.python.org/cpython/raw-file/v3.2.6/Misc/NEWS Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Co-Authored-By: Matthew Oliver <matt@oliver.net.au> Co-Authored-By: Thomas Herve <therve@redhat.com> Change-Id: I388fd697ec88476024b0e9f1ae75ba35ff765282

2015年06月26日 08:54:22 +00:00

Add extra_header_count to document and config. Change-Id: Iec86b488d71553c295afe7098822ce2046df9546

2015年08月05日 22:09:40 +08:00

#extra_header_count = 0

Increase httplib._MAXHEADERS Python 2.7.9+ and 3.2.6+ limits the number of maximum headers in httplib to 100 [1,2,3]. This setting is too low for Swift. By default the maximum number of allowed headers depends on the number of max allowed metadata settings plus a default value of 32 for regular http headers. If for some reason this is not enough (custom middleware for example) it can be increased with the extra_header_count constraint. [1] https://bugs.python.org/issue16037 [2] https://hg.python.org/cpython/raw-file/15c95b7d81dc/Misc/NEWS [3] https://hg.python.org/cpython/raw-file/v3.2.6/Misc/NEWS Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Co-Authored-By: Matthew Oliver <matt@oliver.net.au> Co-Authored-By: Thomas Herve <therve@redhat.com> Change-Id: I388fd697ec88476024b0e9f1ae75ba35ff765282

2015年06月26日 08:54:22 +00:00

swift constraints are now settable via config Change previously hard-coded constants into config variables. This allows deployers to tune their cluster more specifically based on their needs. For example, a deployment that uses direct swift access for public content may need to set a larger header value constraint to allow for the full object name to be represented in the Content- Disposition header (which browsers check to determine the name of a downloaded object). The new settings are set in the [swift-constraints] section of /etc/swift/swift.conf. Comments were also added to this config file. Cleaned up swift/common/constraints.py to pass pep8 1.3.3 Funtional tests now require constraints to be defined in /etc/test.conf or in /etc/swift/swift.conf (in the case of running the functional tests against a local swift cluster). To have any hope of tests passing, the defined constraints must match the constraints on the tested cluster. Removed a ton of "magic numbers" in both unit and functional tests. Change-Id: Ie4588e052fd158314ddca6cd8fca9bc793311465

2012年09月05日 20:49:50 -07:00

# max_object_name_length is the max number of bytes in the utf8 encoding

# of an object name

#max_object_name_length = 1024

# container_listing_limit is the default (and max) number of items

# returned for a container listing request

#container_listing_limit = 10000

# account_listing_limit is the default (and max) number of items returned

# for an account listing request

#account_listing_limit = 10000

# max_account_name_length is the max number of bytes in the utf8 encoding

# of an account name

#max_account_name_length = 256

# max_container_name_length is the max number of bytes in the utf8 encoding

# of a container name

#max_container_name_length = 256

Check if REST API version is valid Swift doesn't check if the used API version is valid. Currently there is only one valid REST API version, but that might change in the future. This patch enforces "v1" or "v1.0" as the version string when accessing account, containers and objects. The list of accepted version strings can be manually overridden using a comma-separated list in swift.conf to make this backward-compatible. The constraint loader has been modified slightly to accept strings as well as integers. Any request to an account, container, and object which does not provide the correct version string will get a 400 BadRequest response. The allowed api versions are by default excluded from /info. Co-Authored-By: Christian Schwede <christian.schwede@enovance.com> Co-Authored-By: John Dickinson <me@not.mn> Closes Bug #1437442 Change-Id: I5ab6e236544378abf2eab562ab759513d09bc256

2015年03月27日 15:50:38 -07:00

# By default all REST API calls should use "v1" or "v1.0" as the version string,

# for example "/v1/account". This can be manually overridden to make this

# backward-compatible, in case a different version string has been used before.

# Use a comma-separated list in case of multiple allowed versions, for example

# valid_api_versions = v0,v1,v2

# This is only enforced for account, container and object requests. The allowed

# api versions are by default excluded from /info.

# valid_api_versions = v1,v1.0

Deprecate per-service auto_create_account_prefix If we move it to constraints it's more globally accessible in our code, but more importantly it's more obvious to ops that everything breaks if you try to mis-configure different values per-service. Change-Id: Ib8f7d08bc48da12be5671abe91a17ae2b49ecfee

2019年12月31日 13:48:13 -06:00

# The prefix used for hidden auto-created accounts, for example accounts in

# which shard containers are created. It defaults to '.'; don't change it.

# auto_create_account_prefix = .

openstack/swift - swift - OpenDev: Free Software Needs Free Tools

212 lines

8.5 KiB

Plaintext

Raw Normal View History

openstack/swift - swift - OpenDev: Free Software Needs Free Tools

212 lines 8.5 KiB Plaintext Raw Normal View History Unescape Escape

212 lines

8.5 KiB

Plaintext

Raw Normal View History