Network Working Group
INTERNET-DRAFT
Expires in: April 2004
Scott Poretsky
Quarry Technologies
Shankar Rao
Qwest Communications
Ray Piatt
Cable and Wireless
October 2003
Framework for Accelerated Stress Benchmarking
<draft-ietf-bmwg-acc-bench-framework-00.txt>
Status of this Memo
This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other
documents at any time. It is inappropriate to use Internet-Drafts
as reference material or to cite them other than as "work in
progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
Abstract
This document provides a framework for executing the Accelerated
Stress Benchmarking. It is intended that this framework be applied
with the Terminology document when using the Methodology document.
Discussion to specify and apply Startup Conditions, Configuration
Sets, and Instability Conditions is provided with examples. The
motivation and benefits of stress testing are also discussed.
Poretsky, Rao, Piatt [Page 1]
INTERNET-DRAFT Framework for Accelerated Stress Benchmarking October 2003
Table of Contents
1. Introduction ................................................ 2
2. Existing definitions ........................................ 2
3. Motivation for Accelerated Stress Benchmarking............... 2
4. Application of Configuration Sets............................ 3
5. Application of Startup Conditions............................ 5
6. Application of Instability Conditions........................ 6
7. Service Provider Application of Accelerated Stress Testing... 6
8. Security Considerations...................................... 6
9. References................................................... 6
10. Author's Address............................................ 6
11. Full Copyright Statement.................................... 7
1. Introduction
This document provides the motivation and framework to perform
Accelerated Stress Benchmarking. The terminology to be used
for Accelerated Stress Benchmarking is defined in [1] and the
methodology is provided in [2]. This document discusses how to
apply the terminology to the benchmarking for producing effective
reproducible tests. Configuration Sets, Startup Conditions, and
Instability Conditions are defined [1] and examples are provided
in this document.
2. Existing definitions
RFC 1242 "Benchmarking Terminology for Network Interconnect
Devices" and RFC 2285 "Benchmarking Terminology for LAN Switching
Devices" should be consulted before attempting to make use of this
document.
For the sake of clarity and continuity this RFC adopts the template
for definitions set out in Section 2 of RFC 1242. Definitions are
indexed and grouped together in sections for ease of reference.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in
this document are to be interpreted as described in RFC 2119.
3. Motivation for Accelerated Stress Benchmarking
Router testing benchmarks have consistently been made in a
monolithic fashion in which a single protocol or behavior is
measured in an isolated environment. It is important to know the
limits for a router/switch's (hereby referred to as Router) behavior
for each protocol, however this does not produce a reliable benchmark
of the router's behavior in a deployed network. Routers in an
operational network are simultaneously configured with multiple
protocols and security policies while forwarding traffic and being
managed.
To accurately benchmark a router for deployment it is necessary to
test that router in operational conditions by simultaneously
configuring the network protocols and security policies, sourcing
traffic, and managing the router. The benchmarks are externally
Poretsky, Rao, Piatt [Page 2]
INTERNET-DRAFT Framework for Accelerated Stress Benchmarking October 2003
observable as control plane or data plane errors at the DUT. It is
helpful to accelerate these network operational conditions so that
the DUT can be benchmarked with faster test duration. Accelerated
Stress Testing of routers provides the following benefits:
1. Evaluation of multiple protocols enabled simultaneously as
configured in deployed networks
2. Evaluation of System and Software Stability
3. Evaluation of Manageability under stressful conditions
4. Identification of Software Coding bugs such as:
a. Memory Leaks
b. Suboptimal CPU Utilization
c. Coding Logic
These benefits produce three advantages for netowrk operations:
1. Increased stability of routers and protocols
2. Hardened routers to DoS attacks
3. Verified manageability under stress
4. Application of Configuration Sets
Configuration Sets are defined in [1] for the Control Plane, Data
Plane, Management Plane, and Security Plane. It is intended that
the user of these documents specify the specific parameters of the
Configuration Set based upon applicability to the device and
network. Example Configuration Sets are provided below.
4.1 Control Plane Configuration Sets
Key protocols for the Control Plane are Routing Protocols, MPLS
Signaling Protocols, and Multicast Protocols. Examples for these
are as follow:
Example Routing Protocol Configuration Set-
PARAMETER UNITS
BGP Enabled/Disabled
Number of EBGP Peers Peers
Number of IBGP Peers Peers
Number of BGP Route Instances Routes
Number of BGP Installed Routes Routes
MBGP Enabled/Disabled
Number of MBGP Route Instances Routes
Number of MBGP Installed Routes Routes
ISIS Enabled/Disabled
ISIS-TE Enabled/Disabled
Number of ISIS Adjacencies Adjacencies
Number of ISIS Routes Routes
Number of Nodes per Area Nodes
Poretsky, Rao, Piatt [Page 3]
INTERNET-DRAFT Framework for Accelerated Stress Benchmarking October 2003
OSPF Enabled/Disabled
OSPF-TE Enabled/Disabled
Number of OSPF Adjacencies Adjacencies
Number of OSPF Routes Routes
Number of Nodes per Area Nodes
Example MPLS Protocol Configuration Set-
PARAMETER UNITS
MPLS-TE
Number of Ingress Tunnels Tunnels
Number of Mid-Point Tunnels Tunnels
Number of Egress Tunnels Tunnels
LDP
Number of Sessions Sessions
Number of FECs FECs
Example Multicast Protocol Configuration Set-
PARAMETER UNITS
PIM-SM Enabled/Disabled
RP Enabled/Disabled
Number of Multicast Groups Groups
MSDP Enabled/Disabled
4.2 Data Plane Configuration Set
The Data Plane Configuration Set includes the Traffic Profile
as defined in [1]. The example configuration set is as follows:
Example Data Plane Configuration Set-
PARAMETER UNITS
Traffic Forwarding Enabled/Disabled
Aggregate Offered Load bps (or pps)
Number of Ingress Interfaces number
Number of Ingress Interfaces number
TRAFFIC PROFILE
Packet Size(s) bytes
Packet Rate(interface) array of packets per second
Number of Flows number
Encapsulation(flow) array of encapsulation type
Poretsky, Rao, Piatt [Page 4]
INTERNET-DRAFT Framework for Accelerated Stress Benchmarking October 2003
4.3 Management Configuration Set
The Management Configuration Set can include SNMP, Logging, Debug,
Telnet, FTP, SSH, and RADIUS parameters. An example is as follows:
Example Management Configuration Set-
PARAMETER UNITS
SNMP GET Rate SNMP Gets/minute
Logging Enabled/Disabled
Protocol Debug Enabled/Disabled
Telnet Rate Sessions/Hour
FTP Rate Sessions/Hour
Concurrent Telnet Sessions Sessions
Concurrent FTP Session Sessions
Packet Statistics Collector Enabled/Disabled
Statistics Sampling Rate X:1 packets
4.4 Security Configuration Set
The Security Configuration Set can include Packet Filters and
Access session restrictions. An example is as follows:
Example Security Configuration Set -
PARAMETER UNITS
Packet Filters Enabled/Disabled
Number of Filters For-Me number
Number of Filter Rules For-Me number
Number of Traffic Filters number
Number of Traffic Filter Rules number
SSH Enabled/Disabled
Number of simultaneous SSH sessions number
RADIUS Enabled/Disabled
TACACS Enabled/Disabled
5. Application of Startup Conditions
Startup conditions are the conditions that must be met in order
for Accelerated Stress benchmarking to begin. Startup Conditions
specify how a particular Configuration Set should be obtained.
Example Startup Conditions include:
PARAMETER UNITS
Routing Session Establishment Rate sessions per minute
User Config Session Establishment Rate number per minute
Security Session Establishment Rate number per minute
Routes Learned Rate routes per minute
MPLS LSPs Establishment Rate number per minute
Poretsky, Rao, Piatt [Page 5]
INTERNET-DRAFT Framework for Accelerated Stress Benchmarking October 2003
6. Application of Instability Conditions
Test conditions that occur during the Accelerated Stress Test
should simulate instability in an operational network.
Repeating these conditions should stress the SUT. Example
Instability Conditions are provided below:
PARAMETER UNITS
Interface Shutdown Cycling Rate interfaces per minute
BGP Session Loss Rate sessions per minute
BGP Route Flap Rate routes per minutes
IGP Route Flap Rate routes per minutes
Route Convergence from Better Next-Hop routes per minutes
LSP Reroute Rate LSP per minute
Overloaded Links number
Amount Links Overloaded % of bandwidth
FTP Rate Mb/minute
IPsec Session Loss sessions per minute
Filter Policy Changes policies per minute
SSH Session Re-Start SSH sessions per minute
7. Accelerated Stress Benchmarking Application
The Accelerated Stress Benchmarking test can be applied in
service provider test environments to benchmark DUTs under
stress in an environment that is reflective of an operational
network. A particular Configuration Set is defined and the
DUT is benchmarked using this and the Instability Conditions.
Varying ConfigurationSets and/or Instability Conditions for
repeated iterations can provide a characterization of the DUT
to help determine future network deployments.
8. Security Considerations
Documents of this type do not directly effect the security of
the Internet or of corporate networks as long as benchmarking
is not performed on devices or systems connected to operating
networks.
9. References
[1] Poretsky, Scott, Rao, Shankar, and Piatt, Ray, "Terminology for
Accelerated Stress Benchmarking, draft-ietf-bmwg-acc-bench-term-
01, work in progress, October 2003.
10. Author's Address
Scott Poretsky
Quarry Technologies
8 New England Executive Park
Burlington, MA 01803
USA
Phone: + 1 781 395 5090
EMail: sporetsky@quarrytech.com
Poretsky, Rao, Piatt [Page 6]
INTERNET-DRAFT Framework for Accelerated Stress Benchmarking October 2003
Shankar Rao
950 17th Street
Suite 1900
Qwest Communications
Denver, CO 80210
USA
Phone: + 1 303 437 6643
Email: srao@qwest.net
Ray Piatt
Cable and Wireless
11700 Plaza America Drive
Reston, VA 20190
USA
Phone: + 1 703 292 2113
Email: rpiatt@cw.net
11. Full Copyright Statement
Copyright (C) The Internet Society (1998). All Rights
Reserved.
This document and translations of it may be copied and
furnished to others, and derivative works that comment on or
otherwise explain it or assist in its implementation may be
prepared, copied, published and distributed, in whole or in
part, without restriction of any kind, provided that the above
copyright notice and this paragraph are included on all such
copies and derivative works. However, this document itself may
not be modified in any way, such as by removing the copyright
notice or references to the Internet Society or other Internet
organizations, except as needed for the purpose of developing
Internet standards in which case the procedures for copyrights
defined in the Internet Standards process must be followed, or
as required to translate it into languages other than English.
The limited permissions granted above are perpetual and will
not be revoked by the Internet Society or its successors or
assigns. This document and the information contained herein is
provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE
INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY
THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY
RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS
FOR A PARTICULAR PURPOSE.
Poretsky, Rao, Piatt [Page 7]