Internet Draft C. Lonvick Document: draft-ietf-syslog-syslog-06.txt Cisco Systems Expires: August, 2001 February 2001 Syslog Protocol Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This work is a product of the IETF syslog Working Group. More information about this effort may be found at http://www.ietf.org/html.charters/syslog-charter.html Comments about this draft should be directed to the syslog working group at the mailing list of syslog-sec@employees.org. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. Copyright Notice Copyright (C) The Internet Society (2001). All Rights Reserved. Abstract This draft describes the observed behavior of the syslog protocol. This protocol has been used for the transmission of event notification messages across networks for many years. Lonvick Expires July 2001 [1] syslog Protocol February 2001 Table of Contents Status of this Memo................................................1 Copyright Notice...................................................1 Abstract...........................................................1 1. Introduction....................................................3 1.1 Events and Generated Messages..................................4 1.2 Operations of the Message Receivers............................5 2. Transport Layer Protocol........................................5 3. Definitions and Architecture....................................6 4. Packet Format and Contents......................................8 4.1 PRI Part of a syslog Packet....................................8 4.2 MSG Part of a syslog Packet...................................10 4.2.1 Original syslog Packets.....................................11 4.2.2 Relayed syslog Packets......................................11 4.2.3 Formats of the Fields in the MSG............................12 5. Conventions....................................................13 5.1 Dates and Times...............................................13 5.2 Domain Name and Address.......................................13 5.3 Originating Process Information...............................14 5.4 Examples......................................................14 6. Security Considerations........................................16 6.1 Packet Parameters.............................................16 6.2 Message Authenticity..........................................17 6.3 Sequenced Delivery............................................18 6.3.1 Single Source to a Destination..............................18 6.3.2 Multiple Sources to a Destination...........................19 6.3.3 Multiple Sources to Multiple Destinations...................19 6.3.4 Replaying...................................................19 6.4 Reliable Delivery.............................................20 6.5 Message Integrity.............................................20 6.6 Message Observation...........................................20 6.7 Message Prioritization and Differentiation....................20 6.8 Misconfiguration..............................................22 6.9 Forwarding Loop...............................................22 6.10 Load Considerations..........................................22 7. Conclusion and Other Efforts...................................22 Acknowledgements..................................................23 References........................................................24 Author's Addresses................................................24 Full Copyright Statement..........................................25 Lonvick Expires August 2001 [2] syslog Protocol February 2001 1. Introduction Since the beginning, life has relied upon the transmission of messages. For the self-aware organic unit, these messages can relay many different things. The messages may signal danger, the presence of food or the other necessities of life, and many other things. In many cases, these messages are informative to other units and require no acknowledgement. As people interacted and created processes, this same principle was applied to societal communications. As an example, severe weather warnings may be delivered through any number of channels - a siren blowing, warnings delivered over television and radio stations, and even through the use of flags on ships. The expectation is that people hearing or seeing these warnings would realize their significance and take appropriate action. In most cases, no responding acknowledgement of receipt of the warning is required or even desired. Along these same lines, operating systems, processes and applications were written to send messages of their own status, or messages to indicate that certain events had occurred. These event messages generally had local significance to the machine operators. As the operating systems, processes and applications grew ever more complex, systems were devised to categorize and log these diverse messages and allow the operations staff to more quickly differentiate the notifications of problems from simple status messages. The syslog process was one such system that has been widely accepted in many operating systems. Flexibility was designed into this process so the operations staff have the ability to configure the destination of messages sent from the processes running on the device. In one dimension, the events that were received by the syslog process could be logged to different files and also displayed on the console of the device. In another dimension, the syslog process could be configured to forward the messages across a network to the syslog process on another machine. The syslog process had to be built network-aware for some modicum of scalability since it was known that the operators of multiple systems would not have the time to access each system to review the messages logged there. The syslog process running on the remote devices could therefore be configured to either add the message to a file, or to subsequently forward it to another machine. In its most simplistic terms, the syslog protocol provides a transport to allow a machine to send event notification messages across IP networks to event message collectors -also known as syslog servers. Since each process, application and operating system was written somewhat independently, there is little uniformity to the content of syslog messages. For this reason, no assumption is made upon the formatting or contents of the messages. The protocol is simply designed to transport these event messages. In all cases, there is one device that originates the message. The syslog process on that machine may send the message to a collector. No acknowledgement of the receipt is made. Lonvick Expires August 2001 [3] syslog Protocol February 2001 One of the fundamental tenants of the syslog protocol and process is its simplicity. No stringent coordination is required between the transmitters and the receivers. Indeed, the transmission of syslog messages may be started on a device without a receiver being configured, or even actually physically present. Conversely, many devices will most likely be able to receive messages without explicit configuration or definitions. This simplicity has greatly aided the acceptance and deployment of syslog. 1.1 Events and Generated Messages The writers of the operating systems, processes and applications have had total control over the circumstances that would generate any message. In some cases, messages are generated to give status. These can be either at a certain period of time, or at some other interval such as the invocation or exit of a program. In other cases, the messages may be generated due to a set of conditions being met. In those cases, either a status message or a message containing an alarm of some type may be generated. It was considered that the writers of the operating systems, processes and applications would quantify their messages into one of several broad categories. These broad categories generally consist of the facility that generated them, along with an indication of the severity of the message. This was so that the operations staff could selectively filter the messages and be presented with the more important and time sensitive notifications quickly, while also having the ability to place status or informative messages in a file for later perusal. Other options for displaying or storing messages have been seen to exist as well. Devices MUST be configured with rules for displaying and/or forwarding the event messages. The rules that have been seen are generally very flexible. An administrator may want to have all messages stored locally as well as to have all messages of a high severity forwarded to another device. They may find it appropriate to also have messages from a particular facility sent to some or all of the users of the device and displayed on the system console. However the administrator decides to configure the disposition of the event messages, the process of having them sent to a syslog collector generally consists of deciding which facility messages and which severity levels will be forwarded, and then defining the remote receiver. For example, an administrator may want all messages that are generated by the mail facility to be forwarded to one particular event message collector. Then the administrator may want to have all kernel generated messages sent to a different syslog receiver while, at the same time, having the critically severe messages from the kernel also sent to a third receiver. It may also be appropriate to have those messages displayed on the system console as well as being mailed to some appropriate people, while at the same time, being sent to a file on the local disk of the device. Conversely, it may be appropriate to have messages from a locally defined process only displayed on the console but not Lonvick Expires August 2001 [4] syslog Protocol February 2001 saved or forwarded from the device. In any event, the rules for this will have to be generated on the device. Since the administrators will then know which types of messages will be received on the collectors, they should then make appropriate rules on those syslog servers as well. The contents of a message have also been at the discretion of its creator. It has been considered to be good form to write the messages so that they are informative to the person who may be reading them. It has also been considered good practice to include a timestamp and some indication of the sending device and the process that originated it in the messages. However, none of those are stringently required. It should be assumed that any process on any device might generate an event message. This may include processes on machines that do not have any local storage - e.g. printers, routers, hubs, switches, and diskless workstations. In that case, it may be imperative that event messages are transported to a collector so that they may be recorded and hopefully viewed by an operator. 1.2 Operations of the Message Receivers It is beyond the scope of this Internet Draft to specify how event messages should be processed when they are received. Like the operations described in Section 1.1, they generally may be displayed to the appropriate people, saved onto disk, further forwarded, or any combination of these. The rules for determining the disposition of received messages have been seen to be identical to the rules for determining the disposition of locally generated messages. As a very general rule, there are usually many devices sending messages to relatively fewer collectors. This fan-in process allows an administrator to aggregate messages into relatively few repositories. 2. Transport Layer Protocol syslog uses the user datagram protocol (UDP) [1] as its underlying transport layer mechanism. The UDP port that has been assigned to syslog is 514. It is RECOMMENDED that the source port also be 514 to indicate that the message is from the syslog process of the sender, but there have been cases seen where valid syslog messages have come from a sender with a source port other than 514. If the sender uses a source port other than 514 then it is RECOMMENDED and has been considered to be good form that subsequent messages are from a single consistent port. Lonvick Expires August 2001 [5] syslog Protocol February 2001 3. Definitions and Architecture The following definitions will be used in this document. A machine that can generate a message will be called a "device". A machine that can receive the message and forward it to another machine will be called a "relay". A machine that receives the message and does not relay it to any other machines will be called a "collector". This has been commonly known as a "syslog server". Any device or relay will be known as the "sender" when it sends a message. Any relay or collector will be known as the "receiver" when it receives the message. The architecture of the devices may be summarized as follows: Devices send messages to relays or collectors with no knowledge of whether it is a collector or relay. Devices and relays may be configured to send the same message to multiple receivers. Lonvick Expires August 2001 [6] syslog Protocol February 2001 The following architectures shown in Diagram 1 are valid while the first one has been known to be the most prevalent. Other arrangements of these examples are also acceptable. +------+ +---------+ |Device|---->----|Collector| +------+ +---------+ +------+ +-----+ +---------+ |Device|---->----|Relay|---->----|Collector| +------+ +-----+ +---------+ +------+ +-----+ +-----+ +---------+ |Device|-->--|Relay|-->--..-->--|Relay|-->--|Collector| +------+ +-----+ +-----+ +---------+ +------+ +-----+ +---------+ |Device|---->----|Relay|---->----|Collector| | |-\ +-----+ +---------+ +------+ \ \ +-----+ +---------+ \-->--|Relay|---->----|Collector| +-----+ +---------+ +------+ +---------+ |Device|---->----|Collector| | |-\ +---------+ +------+ \ \ +-----+ +---------+ \-->--|Relay|---->----|Collector| +-----+ +---------+ +------+ +-----+ +---------+ |Device|---->----|Relay|---->-------|Collector| | |-\ +-----+ /--| | +------+ \ / +---------+ \ +-----+ / \-->--|Relay|-->--/ +-----+ Diagram 1. Some Possible syslog Architectures Lonvick Expires August 2001 [7] syslog Protocol February 2001 4. Packet Format and Contents The syslog packet has two parts. The first part is the PRI and the second part is the MSG. The PRI has three, four, five, or six characters. The MSG will fill the remainder of the syslog packet. There is no ending delimiter but the total length of the packet MUST be 1024 bytes or less. There is no minimum length of the MSG although sending a syslog packet with no contents is worthless and SHOULD NOT be done. The MSG part of the packet has additional fields that are described in Section 4.2 below. 4.1 PRI Part of a syslog Packet The PRI part starts with a leading "<" ('less-than' character), followed by a number, which is followed by a ">" ('greater-than' character). The code set used in this part MUST be seven-bit ASCII in an eight-bit field as described in RFC 2234 [2]. These are the ASCII codes as defined in "USA Standard Code for Information Interchange" [3]. In this, the "<" character is defined as the Augmented Backus-Naur Form (ABNF) %d60, and the ">" character has ABNF value %d62. The number contained within these angle brackets is known as the Priority code and represents both the Facility and Severity as described below. The Priority code consists of one, two, or three decimal integers (ABNF DIGITS) using values of %d48 (for "0") through %d57 (for "9"). Lonvick Expires August 2001 [8] syslog Protocol February 2001 The Facilities and Severities of the messages are numerically coded with decimal values. The operating system and some of the daemons and processes have been assigned a Facility parameter. Processes and applications that have not been assigned a Facility, or that have not been configured to use one of the "local use" Facilities SHOULD use the "user" Facility which has the numerical Facility code of decimal 8. All Facilities are shown in the following table along with their numerical code values. Numerical Facility Code 0 kernel messages 8 user-level messages 16 mail system 24 system daemons 32 security/authorization messages (note 1) 40 messages generated internally by syslogd 48 line printer subsystem 56 network news subsystem 64 UUCP subsystem 72 clock daemon (note 2) 80 security/authorization messages (note 1) 88 FTP daemon 96 NTP subsystem 104 log audit (note 1) 112 log alert (note 1) 120 clock daemon (note 2) 128 local use 0 (local0) 136 local use 1 (local1) 144 local use 2 (local2) 152 local use 3 (local3) 160 local use 4 (local4) 168 local use 5 (local5) 176 local use 6 (local6) 184 local use 7 (local7) Table 1. syslog Message Facilities Note 1 - Various operating systems have been found to utilize Facilities 32, 80, 104 and 112 for security/authorization, audit and alert messages which seem to be similar. Note 2 - Various operating systems have been found to utilize both Facilities 72 and 120 for clock (cron/at) messages. Lonvick Expires August 2001 [9] syslog Protocol February 2001 Each message Priority also has a decimal Severity level indicator. These are described in the following table along with their numerical values. Numerical Severity Code 0 Emergency: system is unusable 1 Alert: action must be taken immediately 2 Critical: critical conditions 3 Error: error conditions 4 Warning: warning conditions 5 Notice: normal but significant condition 6 Informational: informational messages 7 Debug: debug-level messages Table 2. syslog Message Severities The Priority code is calculated by summing the numerical values of the codes of the Facility and Severity. For example, a kernel message with a Severity of Emergency would have a Priority code of 0, while a "local use 4" message with a Severity of Notice would have a Priority code of 165. In the PRI part of a syslog message, these values would be placed between the angle brackets as <0> and <165> respectively. 4.2 MSG Part of a syslog Packet The MSG part of the syslog packet MUST contain visible (printing) characters. The code set traditionally and most often used has also been seven-bit ASCII in an eight-bit field like that used in the PRI part. In this code set, the only allowable characters are the ABNF VCHAR values (%d33-126) and spaces (SP value %d32). However, no indication of the code set used within the MSG is required, nor is it expected. Other code sets MAY be used as long as the characters used in the MSG are exclusively visible characters and spaces similar to those described above. The selection of a code set used in the MSG SHOULD be made with thoughts of the intended receiver. A message containing characters in a code set that cannot be viewed by a recipient will yield no information of value to an operator or administrator looking at it. Lonvick Expires August 2001 [10] syslog Protocol February 2001 Any device receiving a syslog packet MUST check that the packet conforms to a specific format. If the format is apparent and the fields valid, then the relay SHALL NOT make any changes to the received packet before sending it onwards. However, if the format is not apparent or if the fields are invalid, then the relay SHALL change the format of the MSG before retransmitting it but only in ways specified in section 4.2.2. 4.2.1 Original syslog Packets A device SHOULD compose a syslog packet with the PRI and the MSG. There are no set requirements on the contents of the MSG as it is originally sent from a device. It is RECOMMENDED that the MSG have the following fields: TIMESTAMP, HOSTNAME, TAG and then the CONTEXT. Space characters MUST follow the TIMESTAMP and HOSTNAME fields. The contents of these fields are described here and their formats are detailed in Section 4.2.3. If the originally formed message has a TIMESTAMP, then it is the local time of the device. If the originally formed message has a HOSTNAME field, then it will contain the hostname, as it knows itself. If it does not have a hostname, then it will contain its own IP address. If a device has multiple IP addresses, then it has been seen to usually use the IP address from which the message is transmitted. If the originally formed message has a TAG value, then that will be the name of the program or process that generated the message. The CONTEXT contains the details of the message. This has traditionally been a freeform message that gives some detailed information of the event. 4.2.2 Relayed syslog Packets When a relay receives a packet, it will check for a valid PRI. If the first character is not a less-than sign, the relay MUST assume that the packet does not contain a valid PRI. If the 3rd, 4th, or 5th character is not a greater-than sign, the relay again MUST assume that the PRI is not valid. If the relay has been configured to forward packets with a Priority value of 14 (User Facility=8 and Informational Severity=6) then the relay MUST insert a PRI with a Priority value of 14 as well as a MSG as described below. The contents of the received packet will be treated as the CONTEXT of the MSG and appended. This new packet will be sent to the next relay or collector. If the relay finds a valid PRI then it will check its internal configuration. Relays MUST be configured to forward syslog packets on the basis of their Priority value. If the relay finds that it is Lonvick Expires August 2001 [11] syslog Protocol February 2001 configured to forward the received packet, then it MUST check for a valid TIMESTAMP as the first field in the MSG. If it finds a valid TIMESTAMP, then it MUST relay the entire packet unchanged. However, if it does not find a valid TIMESTAMP, then it MUST add a TIMESTAMP followed by a space character. It SHOULD additionally add a HOSTNAME and a space character. These fields are described here and detailed in Section 4.2.3. The remainder of the received packet MUST be treated as the CONTEXT field of the MSG and appended. Since the relay would have no way to determine the originating process from the device that originated the message, the TAG value cannot be determined and will not be included. The TIMESTAMP will be the current local time of the relay. The HOSTNAME will be the name of the device, as it is known by the relay. If the name cannot be determined, the IP address of the device will be used. The Domain Name MUST NOT be included in the HOSTNAME field. If the relay adds a TIMESTAMP and HOSTNAME at the front of the MSG then it MUST check that the total length of the packet is still 1024 bytes or less. If the packet has been expanded beyond 1024 bytes, then the relay MUST truncate the packet to be 1024 bytes. This may cause the loss of vital information from the end of original packet. It is for this reason that it is RECOMMENDED that the PRI and MSG parts of originally generated syslog packets contain the values and fields documented in Section 4.2.1. 4.2.3 Formats of the Fields in the MSG The TIMESTAMP field is in the format of "Mmm dd hh:mm:ss" (without the quote marks) where: Mmm is the English language abbreviation for the month of the year with the first character in uppercase and the other two characters in lowercase. The following are the only acceptable values: Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec dd is the day of the month. If the day of the month is less than 10, then it MUST be represented as a space and then the number. hh:mm:ss is the local time. The hour (hh) is represented in a 24-hour format. Valid entries are between 00 and 23, inclusive. The minute (mm) and second (ss) entries are between 00 and 59 inclusive. A space character MUST follow the TIMESTAMP field. Lonvick Expires August 2001 [12] syslog Protocol February 2001 The HOSTNAME field MUST contain the hostname of the device as specified in STD-13 [4]. A space character MUST also follow the hostname to terminate this field. The TAG is a string of ABNF alphanumeric characters that MUST NOT exceed 32 characters. Any non-alphanumeric character will terminate the TAG field and will be assumed to be the starting character of the CONTEXT field. Most commonly, the first character of the CONTEXT field that signifies the conclusion of the TAG field has been seen to be the left square bracket character ("["), a colon character (":"), or a space character. This is explained in more detail in Section 5.3. The CONTEXT field may fill the remainder of the MSG field. 5. Conventions Although Section 4 of this document specifies all requirements for the syslog protocol format and contents, certain conventions have come about over time for the inclusion of additional information within the syslog message. It must be plainly stated that these items are not mandated but may be included for completeness and to give the recipient some additional clues of their origin and nature. 5.1 Dates and Times It has been found that some network administrators like to archive their syslog messages over long periods of time. For the convenience of these people and for automated message parsers, a more explicit time stamp has been seen to have been added to some messages. Some devices will send an original syslog message with a 2 character or 4 character year field immediately after the space terminating the TIMESTAMP. This is not consistent with the original intent of the order and format of the fields. If implementers wish to contain a more specific date and time stamp within the message, it should be within the CONTEXT field. Implementers may wish to utilize the ISO 8601 [5] date and time formats if they want to include more explicit date and time information. 5.2 Domain Name and Address To readily identify the device that originated the message, it may be a good practice to include its fully qualified domain name (FQDN) and its IP address within the CONTEXT field. Traditionally, however, only the hostname has been included in the HOSTNAME field. Lonvick Expires August 2001 [13] syslog Protocol February 2001 5.3 Originating Process Information It has also been considered to be a good practice to include some information about the process on the device that generated the message - if that concept exists. This is usually the process name and process id (often known as the "pid") for robust operating systems. The process name is commonly displayed in the TAG field. Quite often, additional information is included at the beginning of the CONTEXT field. The format of "TAG[pid]:" - without the quote marks - is common. The left square bracket is used to terminate the TAG field in this case and is then the first character in the CONTEXT field. If the process id is immaterial, it may be left off. In that case, a colon and a space character usually follow the TAG. This would be displayed as "TAG: " without the quotes. In that case, the colon is the first character in the CONTEXT field. 5.4 Examples As examples, these are valid messages as they may be observed on the wire between two devices. In the following examples, each message starts with the less-than character but has been indented, with line breaks inserted in this document for readability. Example 1 <34>Oct 11 22:14:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8 This example shows an authentication error in an attempt to acquire additional privileges. It also shows the command attempted and the user attempting it. This was recorded as an original message from the device called mymachine. A relay receiving this would not make any changes before sending it along as it contains a properly formatted PRI part and TIMESTAME field in the MSG part. The TAG value in this example is the process "su". The colon has terminated the TAG field and is the first character of the CONTEXT field. In this case, the process id (pid) would be considered transient and anyone looking at this syslog message would gain no useful information from knowing the pid. It has not been included so the first two characters of the CONTEXT field are the colon and a space character. Example 2 <14>Use the BFG! While this is a valid message, it has extraordinarily little useful information. It does not contain a timestamp or any indication of the source of the message. Since it is a user-generated message, it is consistent that it is not associated with a process and may Lonvick Expires August 2001 [14] syslog Protocol February 2001 therefor not have a TAG. If this message is stored on paper or disk, subsequent review of the message will not yield anything of value. This example is obviously an original message from a device. A relay MUST make changes to the message as described in Section 4.2.2 before forwarding it. The resulting relayed message is shown below. <14>Feb 5 17:32:18 10.0.0.99 Use the BFG! In this relayed message, a TIMESTAMP has been added along with a HOSTNAME in the MSG part. Subsequent relays will not make any further changes to this message. It should be noted in this example that the day of the month is less than 10. Since single digits in the date (5 in this case) are preceded by a space in the TIMESTAMP format, there are two spaces following the month in the TIMESTAMP before the day of the month. Also, the relay appears to have no knowledge of the host name of the device sending the message so it has inserted the IP address of the device into the HOSTNAME field. Example 3 <165>Aug 24 05:34:00 CST 1987 mymachine myproc[10]: %% It's time to make the do-nuts. %% Ingredients: Mix=OK, Jelly=OK # Devices: Mixer=OK, Jelly_Injector=OK, Frier=OK # Transport: Conveyer1=OK, Conveyer2=OK # %% This message does have a valid PRI part with a Priority value indicating that it came from a locally defined facility (local4) with a severity of Notice. The MSG part has a proper TIMESTAMP field in the message. A relay will not modify this message before sending it. However, the HOSTNAME and TAG fields are not consistent with the definitions in Section 4.2.1. The HOSTNAME field would be construed to be "CST" and the TAG value would be "1987". It should be noted that the information contained in the CONTEXT of this example is not telemetry data, nor is it supervisory control or data acquisition information. Due to the security concerns listed in Section 6 of this document, information of that nature should probably not be conveyed across this protocol. Example 4 <0>1990 Oct 22 10:52:01 TZ-6 scapegoat.dmz.example.org 10.1.2.3 sched[0]: That's All Folks! This example has a lot of extraneous information throughout. A human or sufficiently adaptable automated parser would be able to determine the date and time information as well as a fully qualified domain name (FQDN) [4] and IP address. The information about the nature of the event is, however, limited. Due to the indicated Lonvick Expires August 2001 [15] syslog Protocol February 2001 severity of the event, the process may not have been able to gather or send anything more informative. It may have been fortunate to have generated and sent this message at all. This example is obviously an original message from a device. Since the first field in the MSG part is not a TIMESTAMP in the format defined in Section 4.2.3, it MUST be modified by a relay. A relay will add a TIMESTAMP and HOSTNAME as follows and will treat the entire received MSG part from the original packet as the CONTEXT field of the new MSG part. <0>Oct 22 10:52:12 scapegoat 1990 Oct 22 10:52:01 TZ-6 scapegoat.dmz.example.org 10.1.2.3 sched[0]: That's All Folks! 6. Security Considerations An odor may be considered to be a message that does not require any acknowledgement. People tend to avoid bad odors but are drawn to odors that they associate with good food. The acknowledgement of the receipt of the odor or scent is not required and indeed it may be the height of discretion to totally ignore some odors. On the other hand, it is usually considered good civility to acknowledge the prowess of the cook merely from the ambiance wafting from the kitchen. Similarly, various species have been found to utilize odors to attract mates. One species of moths use this scent to find each other. However, it has been found that bolas spiders can mimic the odor of the female moths of this species. This scent will then attract male moths, which will follow it with the expectation of finding a mate. Instead, when they arrive at the source of the scent, they will be eaten. [6] This is a case of a false message being sent out with inimical intent. In its local use, the syslog process places event notification messages into files on that system. This relies upon the integrity of the system for the protection of the messages. The subsequent configuration of the syslog process to use the syslog protocol to transport the messages to another collector was an extension of the delivery of event notification messages and it exhibits the same trust of the network. One of the consequences of the fundamental simplicity of syslog is that any device can generate any syslog message and send it to any other machine through the syslog protocol. As such there are some concerns about the applicability of this protocol in situations that require robust delivery. Along the lines of the analogy, computer event messages may be sent accidentally, erroneously and even maliciously. At the time of this writing, however, there have not been any reports of any machine consuming any other machine. 6.1 Packet Parameters Lonvick Expires August 2001 [16] syslog Protocol February 2001 As was described above, the message length MUST NOT exceed 1024 bytes. Attacks have seen where syslog messages are sent to a receiver that have message lengths greater than 1024 bytes. In some older versions of syslog, the receipt of syslog packets that had a message greater than 1024 bytes caused problems. syslog message receivers must not malfunction upon the receipt of packets where the message length is greater than 1024 bytes. If a message receiver does receive a message whose length is greater than 1024 bytes, it may log all of, or the source address and some of the contents of the message, or it may discard the message altogether. Devices MUST NOT retransmit messages whose received length exceeds 1024 bytes. Similarly, the receiver must rigidly enforce the correctness of the message body. syslog collectors must not malfunction if received messages do not have the less-than and greater-than characters around a valid Priority value. They MUST treat these messages as the unformatted CONTEXT as was described in Section 4.2.2 if they relay it. Also, received messages must contain printable text in the message as was described throughout Section 4. Devices must not malfunction if they receive a message containing characters other than the characters described above. 6.2 Message Authenticity The syslog delivery mechanism does not strongly associate the message with the message sender. The receiver of that packet will not be able to ascertain that the message was indeed sent from the reported sender, or if the packet was sent from another device. It should be noted here that the message receiver does not need to verify that the HOSTNAME in the MSG part match the name of the IP address contained in the Source Address field of the IP packet. One possible consequence of this is that a misconfigured machine may send syslog messages to a collector representing itself as another machine. The administrative staff may become confused that the status of the supposed sender of the messages may not be accurately reflected in the received messages. The administrators may not be able to readily discern that there are two or more machines representing themselves as the same machine. It should also be noted that some cases of filling the HOSTNAME field in the MSG part might only have local significance and that may only be ephemeral. If the device had obtained an IP address from a DHCP pool, then any association between an identifier and an actual source would not always hold true. The inclusion of a fully qualified domain name in the CONTEXT may give the administrators the best chance of identifying the source of each message if it can always be associated with an IP address or if it can always be associated with a unique machine. Lonvick Expires August 2001 [17] syslog Protocol February 2001 Malicious exploits of this vulnerability have also been noted. An attacker may transmit syslog messages (either from the machine that the messages purport from which to be sent or from any other machine) to a collector. The attacks that have been noted run along these lines: An attacker may hide the true nature of an attack amidst many other messages. As an example, an attacker may start generating messages indicating a problem on some machine. This may get the attention of the system administrators who will spend their time investigating the alleged problem. During this time, the attacker may be able to compromise a different machine, or a different process on the same machine. An attacker may generate false syslog messages to give untrue indications of status or of events. As an example, an attacker may stop a critical process on a machine, which may generate a notification of exit. The attacker may subsequently generate a false notification that the process had been restarted. The system administrators may accept that misinformation and not verify that the process had indeed been restarted. 6.3 Sequenced Delivery As a general rule, the forensics of a network anomaly rely upon reconstructing the sequence of events. In a perfect world, the messages would be received on the syslog collector in the order of their generation from the other devices and anyone looking at these records would have an accurate picture of the sequence of events. Unfortunately, the syslog process and protocol do not ensure ordered delivery. This section details some of the problems that may be encountered from this. 6.3.1 Single Source to a Destination The syslog records are usually presented (placed in a file, displayed on the console, etc.) in the order in which they are received. This is not always in accordance with the sequence in which they were generated. As they are transported across an IP network, some out of order receipt should be expected. This may lead to some confusion as messages may be received that would indicate that a process has stopped before it was started. This may be somewhat rectified if the originating process had timestamped or numbered each of the messages before transmission. In this, the sending device should utilize an authoritative time source. It should be remembered, however, that not all devices are capable of receiving time updates, and not all devices can timestamp their messages. Lonvick Expires August 2001 [18] syslog Protocol February 2001 6.3.2 Multiple Sources to a Destination In syslog, there is no concept of unified event numbering. Single devices are free to include a sequence number within the CONTEXT but that can hardly be coordinated between multiple devices. In such cases, multiple devices may report that each one is sending message number one. Again, this may be rectified somewhat if the sending devices utilize a timestamp from an authoritative source in their messages. As has been noted, however, even messages from a single device to a single collector may be received out of order. This situation is compounded when there are several devices configured to send their syslog messages to a single collector. Messages from one device may be delayed so the collector receives messages from another device first even though the messages from the first device were generated before the messages from the second. If there is no timestamp or coordinated sequence number, then the messages may be presented in the order in which they were received which may give an inaccurate view of the sequence of actual events. 6.3.3 Multiple Sources to Multiple Destinations The plethora of configuration options available to the network administrators may further skew the perception of the order of events. It is possible to configure a group of devices to send the status messages -or other informative messages- to one collector, while sending messages of relatively higher importance to another collector. Additionally, the messages may be sent to different files on the same collector. If the messages do not contain timestamps from the source, it may be difficult to order the messages if they are kept in different places. An administrator may not be able to determine if a record in one file occurred before or after a record in a different file. This may be somewhat alleviated by placing marking messages with a timestamp into all destination files. If these have coordinated timestamps, then there will be some indication of the time of receipt of the individual messages. 6.3.4 Replaying Without any sequence indication or timestamp, messages may be recorded and replayed at a later time. An attacker may record a set of messages that indicate normal activity of a machine. At a later time, that attacker may remove that machine from the network and replay the syslog messages to the collector. Even with a TIMESTAMP field in the MSG part, an attacker may record the packets and could simply modify them to reflect the current time before retransmitting them. The administrators may find nothing unusual in the received messages and their receipt would falsely indicate normal activity of the machine. Lonvick Expires August 2001 [19] syslog Protocol February 2001 6.4 Reliable Delivery As there is no mechanism within either the syslog process or the protocol to ensure delivery, and since the underlying transport is UDP, some messages may be lost. They may either be dropped through network congestion, or they may be maliciously intercepted and discarded. The consequences of the drop of one or more syslog messages cannot be determined. If the messages are simple status updates, then their non-receipt may either not be noticed, or it may cause an annoyance for the system operators. On the other hand, if the messages are more critical, then the administrators may not become aware of a developing and potentially serious problem. Messages may also be intercepted and discarded by an attacker as a way to hide unauthorized activities. 6.5 Message Integrity Besides being discarded, syslog messages may be damaged in transit, or an attacker may maliciously modify them. In the case of a packet containing a syslog message being damaged, there are various mechanisms built into the link layer as well as into the IP [7] and UDP protocols which may detect the damage. An intermediary router may discard a damaged IP packet [8]. Damage to a UDP packet may be detected by the receiving UDP module, which may silently discard it. In any case, the original contents of the message will not be delivered to the collector. Additionally, if an attacker is positioned between the sender and collector of syslog messages, they may be able to intercept and modify those messages while in-transit to hide unauthorized activities. 6.6 Message Observation While there are no strict guidelines pertaining to the event message format, most syslog messages are generated in human readable form with the assumption that capable administrators should be able to read them and understand their meaning. Neither the syslog protocol nor the syslog application have mechanisms to provide confidentiality of the messages in transit. In most cases passing clear-text messages is a benefit to the operations staff if they are sniffing the packets off of the wire. The operations staff may be able to read the messages and associate them with other events seen from other packets crossing the wire to track down and correct problems. Unfortunately, an attacker may also be able to observe the human-readable contents of syslog messages. The attacker may then use the knowledge gained from those messages to compromise a machine or do other damage. 6.7 Message Prioritization and Differentiation Lonvick Expires August 2001 [20] syslog Protocol February 2001 While the processes that create the messages may signify the importance of the events through the use of the message Priority value, there is no distinct association between this value and the importance of delivery of the packet. As an example of this, consider an application that generates two event messages. The first is a normal status message but the second could be an important message denoting a problem with the process. This second message would have an appropriately higher Severity value associated with the importance of that event. If the operators had configured that both of these messages be transported to a syslog collector then they would, in turn, be given to UDP for transmission. Under normal conditions, no distinction would be made between them and they would be transmitted in their order. Again, under normal circumstances, the receiver would accept syslog messages as they are received. If many devices are transmitting normal status messages, but one is transmitting an important event message, there is no inherent mechanism within the syslog protocol to prioritize the important message over the other messages. On a case-by-case basis, device operators may find some way to associate the different levels with the quality of service identifiers. As an example, the operators may elect to define some linkage between syslog messages that have a specific Priority value with a specific value to be used in the IPv4 Precedence field [7], the IPv6 Traffic Class octet [9], or the Differentiated Services field [10]. In the above example, the operators may have the ability to associate the status message with normal delivery while associating the message indicating a problem with a high reliability, low latency queue as it goes through the network. This would have the affect of prioritizing the essential messages before the normal status messages. Even with this hop-by-hop prioritization, this queuing mechanism could still lead to head of line blocking on the transmitting device as well as buffer starvation on the receiving device if there are many near- simultaneous messages being sent or received. This behavior is not unique to syslog but is endemic to all operations that transmit messages serially. There are security concerns for this behavior. Head of line blocking of the transmission of important event messages may relegate the conveyance of important messages behind less important messages. If the queue is cleared appropriately, this may only add seconds to the transmission of the important message. On the other hand, if the queue is not cleared, then important messages may not be transmitted. Also at the receiving side, if the syslog receiver is suffering from buffer starvation due to large numbers of messages being received near-simultaneously, important messages may be dropped indiscriminately along with other messages. While these are problems with the devices and their capacities, the protocol security concern is that there is no prioritization of the relatively more important messages over the less important messages. Lonvick Expires August 2001 [21] syslog Protocol February 2001 6.8 Misconfiguration Since there is no control information distributed about any messages or configurations, it is wholly the responsibility of the network administrator to ensure that the messages are actually going to the intended recipient. Cases have been noted where devices were inadvertently configured to send syslog messages to the wrong receiver. In many cases, the inadvertent receiver may not be configured to receive syslog messages and it will probably discard them. In certain other cases, the receipt of syslog messages has been known to cause problems for the unintended recipient. [11] If messages are not going to the intended recipient, then they cannot be reviewed or processed. 6.9 Forwarding Loop As it is shown in Figure 1, machines may be configured to relay syslog messages to subsequent relays before reaching a collector. In one particular case, an administrator found that he had mistakenly configured two relays to forward messages with certain Priority values to each other. When either of these machines either received or generated that type of message, it would forward it to the other relay. That relay would, in turn, forward it back. This cycle did cause degradation to the intervening network as well as to the processing availability on the two devices. Network administrators must take care to not cause such a death spiral. 6.10 Load Considerations Network administrators must take the time to estimate the appropriate size of the syslog receivers. An attacker may perform a Denial of Service attack by filling the disk of the collector with false messages. Placing the records in a circular file may alleviate this but that has the consequence of not ensuring that an administrator will be able to review the records in the future. Along this line, a receiver or collector must have a network interface capable of receiving all messages sent to it. Administrators and network planners must also critically review the network paths between the devices, the relays, and the collectors. Generated syslog messages should not overwhelm any of the network links. 7. Conclusion and Other Efforts The syslog protocol may be effectively used to transport event notification messages across a network. It is highly recommended that the network operators who choose to use this understand the characteristics of the protocol and its security implications. Lonvick Expires August 2001 [22] syslog Protocol February 2001 There have been attempts in the past to standardize the format of the syslog message. The most notable attempt culminated in a BOF at the Fortieth Internet Engineering Task Force meeting in 1997. This was the Universal Logging Protocol (ulp) BOF and the minutes of their meeting are on-line at the IETF Proceedings web site. [12] Many good thoughts came from that effort and interested implementers may want to find some of the notes or papers produced from that effort. At the time of this writing, efforts are underway to allow the usage of international character sets in applications that have been traditionally thought of as being text-only. The HOSTNAME and TIMESTAMP fields described above are representative of this. Also, the entire CONTEXT field has traditionally been printing characters and spaces in the code set known as US-ASCII. It is hoped that the proponents of these internationalization efforts will find a suitable way to allow the use of international character sets within syslog messages without being disruptive. It should also be hoped that implementers will allow for the future acceptance of additional code sets and that they may make appropriate plans. Again, it must be cautioned that the simplicity of the existing system has been a tremendous value to its acceptance. Anything that lessens that simplicity may diminish that value. Acknowledgements The following people provided content feedback during the writing of this draft: Jon Knight <J.P.Knight@lboro.ac.uk> Magosanyi Arpad <mag@bunuel.tii.matav.hu> Balazs Scheidler <bazsi@balabit.hu> Jon Callas <jon@counterpane.com> Eliot Lear <lear@cisco.com> Petter Reinholdtsen <pere@hungry.com> Darren Reed <darrenr@reed.wattle.id.au> Alfonso De Gregorio <dira@speedcom.it> Eric Allman <eric@sendmail.com> Andrew Ross <andrew@kiwi-enterprises.com> Eric Allman is the original inventor and author of the syslog daemon and protocol. The author of this draft and the community at large would like to express their appreciation for this work and for the usefulness that it has provided over the years. A large amount of additional information about this de-facto standard operating system feature may usually be found in the syslog.conf file as well as in the man pages for syslog.conf, syslog, syslogd, and logger, of many Unix and Unix-like devices. Lonvick Expires August 2001 [23] syslog Protocol February 2001 References 1 Postel, J., "User Datagram Protocol", STD 6, RFC 768, USC/Information Sciences Institute, August 1980 2 Crocker, D., and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", RFC 2234, November, 1997 3 USA Standard Code for Information Interchange, USASI X3.4-1968 4 Mockapetris, P.V., "Domain names - concepts and facilities", RFC 1034, STD 13, Nov 1987 5 Data elements and interchange formats - Information exchange - Representation of dates and times, International Organization for Standardization, Reference number ISO 8601 : 1988 (E), 1988 6 Stowe, M., et al, "Chemical Mimicry: Bolas Spiders Emit Components of Moth Prey Species Sex Pheromones", Science, 1987 7 Postel, J., "Internet Protocol", STD 5, RFC 791, USC/Information Sciences Institute, September 1981 8 Baker, F., "Requirements for IP Version 4 Routers", RFC 1812, June 1995 9 Deering, S. and R. Hinden, "Internet Protocol, Version 6 (IPv6) Specification", RFC 2460, December 1998 10 Nichols, K., S. Blake, F. Baker, D. Black, "Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers", RFC 2474, December 1998 11 Cisco Systems Product Security Incident Response Team (PSIRT), "Field Notice: Cisco IOS(r) Syslog Crash", January 11, 1999 http://www.cisco.com/warp/public/707/advisory.html 12 Walker, D., IETF Secretariat, "Proceedings of the Fortieth Internet Engineering Task Force, Washington, DC, USA, December 8-12, 1997 http://www.ietf.org/proceedings/97dec/index.html Author's Addresses Chris Lonvick Cisco Systems 12515 Research Blvd. Phone: +1.512.378.1182 Austin, TX, USA Email: clonvick@cisco.com Lonvick Expires August 2001 [24] syslog Protocol February 2001 Full Copyright Statement Copyright (C) The Internet Society (2001). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Lonvick Expires August 2001 [25]