Network Working Group Samuel J. Leffler Request for Comments: 893 Michael J. Karels University of California at Berkeley April 1984 Trailer Encapsulations Status of this Memo This RFC discusses the motivation for use of "trailer encapsulations" on local-area networks and describes the implementation of such an encapsulation on various media. This document is for information only. This is NOT an official protocol for the ARPA Internet community. Introduction A trailer encapsulation is a link level packet format employed by 4.2BSD UNIX (among others). A trailer encapsulation, or "trailer", may be generated by a system under certain conditions in an effort to minimize the number and size of memory-to-memory copy operations performed by a receiving host when processing a data packet. Trailers are strictly a link level packet format and are not visible (when properly implemented) in any higher level protocol processing. This note cites the motivation behind the trailer encapsulation and describes the trailer encapsulation packet formats currently in use on 3 Mb/s Experimental Ethernet, 10 Mb/s Ethernet, and 10 Mb/s V2LNI ring networks [1]. The use of a trailer encapsulation was suggested by Greg Chesson, and the encapsulation described here was designed by Bill Joy. Motivation Trailers are motivated by the overhead which may be incurred during protocol processing when one or more memory to memory copies must be performed. Copying can be required at many levels of processing, from moving data between the network medium and the host's memory, to passing data between the operating system and user address spaces. An optimal network implementation would expect to incur zero copy operations between delivery of a data packet into host memory and presentation of the appropriate data to the receiving process. While many packets may not be processed without some copying operations, when the host computer provides suitable memory management support it may often be possible to avoid copying simply by manipulating the appropriate virtual memory hardware. In a page mapped virtual memory environment, two prerequisites are usually required to achieve the goal of zero copy operations during packet processing. Data destined for a receiving agent must be Leffler & Karels [Page 1]
RFC 893 April 1984 aligned on a page boundary and must have a size which is a multiple of the hardware page size (or filled to a page boundary). The latter restriction assumes virtual memory protection is maintained at the page level; different architectures may alter these prerequisites. Data to be transmitted across a network may easily be segmented in the appropriate size, but unless the encapsulating protocol header information is fixed in size, alignment to a page boundary is virtually impossible. Protocol header information may vary in size due to the use of multiple protocols (each with a different header), or it may vary in size by agreement (for example, when optional information is included in the header). To insure page alignment the header information which prefixes data destined for the receiver must be reduced to a fixed size; this is normally the case at the link level of a network. By taking all (possibly) variable length header information and moving it after the data segment a sending host may "do its best" in allowing the receiving host the opportunity to receive data on a page aligned boundary. This rearrangement of data at the link level to force variable length header information to "trail" the data is the substance of the trailer encapsulation. There are several implicit assumptions in the above argument. 1. The receiving host must be willing to accept trailers. As this is a link level encapsulation, unless a host to host negotiation is performed (preferably at the link level to avoid violating layering principles), only certain hosts will be able to converse, or their communication may be significantly impaired if trailer packets are mixed with non-trailer packets. 2. The cost of receiving data on a page aligned boundary should be comparable to receiving data on a non-page aligned boundary. If the overhead of insuring proper alignment is too high, the savings in avoiding copy operations may not be cost effective. 3. The size of the variable length header information should be significantly less than that of the data segment being transmitted. It is possible to move trailing information without physically copying it, but often implementation constraints and the characteristics of the underlying network hardware preclude merely remapping the header(s). 4. The memory to memory copying overhead which is expected to be performed by the receiver must be significant enough to warrant the added complexity in the both the sending and receiving host software. The first point is well known and the motivation for this note. Leffler & Karels [Page 2]
RFC 893 April 1984 Thought has been given to negotiating the user of trailers on a per host basis using a variant of the Address Resolution Protocol [2] (actually augmenting the protocol), but at present all systems using trailers require hosts sharing a network medium to uniformly accept trailers or never transmit them. (The latter is easily carried out at boot time in 4.2BSD without modifying the operating system source code.) The second point is (to our knowledge) insignificant. While a host may not be able to take advantage of the alignment and size properties of a trailer packet, it should nonetheless never hamper it. Regarding the third point, let us assume the trailing header information is copied and not remapped, and consider the header overhead in the TCP/IP protocols as a representative example [3]. If we assume both the TCP and IP protocol headers are part of the variable length header information, then the smallest trailer packet (generated by a VAX) would have 512 bytes of data and 40+ bytes of header information (plus the trailer header described later). While the trailing header could have IP and/or TCP options included this would normally be rare (one would expect most TCP options, for example, to be included in the initial connection setup exchange) and certainly much smaller than 512 bytes. If the data segment is larger, the ratio decreases and the expected gain due to fewer copies on the receiving end increases. Given the relative overheads of a memory to memory copy operation and that of a page map manipulation (including translation buffer invalidation), the advantage is obvious. The fourth issue, we believe, is actually a non-issue. In our implementation the additional code required to support the trailer encapsulation amounts to about a dozen lines of code in each link level "network interface driver". The resulting performance improvement more than warrants this minor investment in software. It should be recognized that modifying the network (and normal link) level format of a packet in the manner described forces the receiving host to buffer the entire packet before processing. Clever implementations may parse protocol headers as the packet arrives to find out the actual size (or network level packet type) of an incoming message. This allows these implementations to avoid preallocating maximum sized buffers to incoming packets which it can recognize as unacceptable. Implementations which parses the network level format on the fly are violating layering principles which have been extolled in design for some time (but often violated in implementation). The problem of postponing link level type Leffler & Karels [Page 3]
RFC 893 April 1984 recognition is a valid criticism. In the case of network hardware which supports DMA, however, the entire packet is always received before processing begins. Trailer Encapsulation Packet Formats In this section we describe the link level packet formats used on the 3 Mb/s Experimental Ethernet, and 10 Mb/s Ethernet networks as well as the 10 Mb/s V2LNI ring network. The formats used in each case differ only in the format and type field values used in each of the local area network headers. The format of a trailer packet is shown in the following diagram. +----+-------------------------------------------------+----+ | LH | data | TH | +----+-------------------------------------------------+----+ ^ ( ^ ) ^ LH: The fixed-size local network header. For 10 a Mb/s Ethernet, the 16-byte Ethernet header. The type field in the header indicates that both the packet type (trailer) and the length of the data segment. For the 10 Mb/s Ethernet, the types are between 1001 and 1010 hexadecimal (4096 and 4112 decimal). The type is calculated as 1000 (hex) plus the number of 512-byte pages of data. A maximum of 16 pages of data may be transmitted in a single trailer packet (8192 bytes). data: The "data" portion of the packet. This is normally only data to be delivered to the receiving processes (i.e. it contains no TCP or IP header information). Data size is always a multiple of 512 bytes. TH: The "trailer". This is actually a composition of the original protocol headers and a fixed size trailer prefix which defines the type and size of the trailing data. The format of a trailer is shown below. The carats (^) indicate the page boundaries on which the receiving host would place its input buffer for optimal alignment when Leffler & Karels [Page 4]
RFC 893 April 1984 receiving a trailer packet. The link level receiving routine is able to locate the trailer using the size indicated in the link level header's type field. The receiving routine is expected to discard the link level header and trailer prefix, and remap the trailing data segment to the front of the packet to regenerate the original network level packet format. Trailer Format +----------------+----------------+------~...~----------+ | TYPE | HEADER LENGTH | ORIGINAL HEADER(S) | +----------------+----------------+------~...~----------+ Type: 16 bits The type field encodes the original link level type of the transmitted packet. This is the value which would normally be placed in the link level header if a trailer were not generated. Header length: 16 bits The header length field of the trailer data segment. This specifies the length in bytes of the following header data. Original headers: <variable length> The header information which logically belongs before the data segment. This is normally the network and transport level protocol headers. Summary A link level encapsulation which promotes alignment properties necessary for the efficient use of virtual memory hardware facilities has been described. This encapsulation format is in use on many systems and is a standard facility in 4.2BSD UNIX. The encapsulation provides an efficient mechanism by which cooperating hosts on a local network may obtain significant performance improvements. The use of this encapsulation technique currently requires uniform cooperation from all hosts on a network; hopefully a per host negotiation mechanism may be added to allow consenting hosts to utilize the encapsulation in a non-uniform environment. Leffler & Karels [Page 5]
RFC 893 April 1984 References [1] "The Ethernet - A Local Area Network", Version 1.0, Digital Equipment Corporation, Intel Corporation, Xerox Corporation, September 1980. [2] Plummer, David C., "An Ethernet Address Resolution Protocol", RFC-826, Symbolics Cambridge Research Center, November 1982. [3] Postel, J., "Internet Protocol", RFC-791, USC/Information Sciences Institute, September 1981. Leffler & Karels [Page 6]