Home » Fast and Flexible Logging with Vista’s Common Log File System

Fast and Flexible Logging with Vista’s Common Log File System

• By Charlie Frank
• Last updated: March 2, 2007

ogging is a vital capability for enterprise-class applications! For example, consider some of the motivations for examining the logs generated by Internet Information Services (IIS):

• Usability: Which pages are the most popular? Which pages aren’t visitors finding?
• Reliability and supportability: For how long was the site down? How did traffic respond when the site was restored?
• Performance: How many visitors do we have per day? When is traffic the heaviest?
• Security and auditing: What is the IP address of an offending visitor? How many times was a particular document downloaded this week?

Until now, adding logging capabilities to your application meant designing, writing, and testing your own logging framework?and there’s more to logging than meets the eye. A robust logging framework must handle issues such as:

• How to manage available disk space
• How to enforce security
• Support for multiple log entry formats
• Granting (or preventing) access to the log from multiple applications simultaneously

During Vista’s development, Microsoft focused a great deal of its development efforts on providing core application services in the operating system and exposing those services to the .NET framework. With the new Common Log File System (CLFS), you won’t have to reinvent the logging wheel.

The Common Log File System
The Common Log File System (CLFS) is a unified management and storage mechanism for log files. It’s native to Windows Vista and later operating systems, but is also available on Windows Server 2003 R2. It has an impressive list of features:

• Log entry formats are completely customizable.
• Log files can be stored anywhere.
• Log files automatically expand or truncate according to policy settings.
• No need to directly manipulate physical log files.
• Security controlled by the operating system.
• Multiple applications can access the same log.

A CLFS log is an abstract concept that represents a collection of log records. A log record is a unit of application data; it’s the data your application writes to the log when you encounter an unexpected condition or need to trace an activity.

A CLFS log has two components: the log store and the extents. The log store contains metadata information about the log, and the extents are where CLFS physically stores log records. A small log might have two extents, while a large log might have hundreds of extents. CLFS dynamically grows your application’s log by adding extents.

CLFS stores the log store in a physical file called the log file, which is a small file with a .blf extension. CLFS stores each extent in a physical file called a container. Containers are always multiples of 512 KB, so every container in a CLFS log is the same size. It’s no coincidence that containers and log stores are similar to sectors and clusters on a hard disk; Microsoft designed CLFS for high performance.

CLFS stores log records in sequence. There are two kinds of sequences. The first type of sequence is a logical sequence. When you create a log record you can link it to other log records to create logical chains. How you chain log records together is entirely up to you. The second type of sequence is a physical sequence; it’s the ordered set of log records in a container file.

Unmanaged windows applications access CLFS through the Win32 API: Clfsw32.h and Clfsw32.dll. Managed .NET applications access CLFS through the .NET 3.0 System.IO.Log namespace. The Win32 CLFS functions emphasize the physical concepts; for example, you add an extent to a log store by passing a log file handle to the AddLogContainer function. In contrast, the .NET CLFS methods emphasize the abstract concepts; for example, you add an extent to a log store by calling the LogStore.Extents.Add method.

Use the Win32 CLFS API if your application isn’t managed or if you need fine-tuned control over logs. Otherwise, use the .NET CLFS API?it’s a higher level API that’s easier to understand and use. Using the .NET CLFS API also allows you to take advantage of .NET’s robust serialization support, which is useful for reading and writing log records.

The remainder of this article focuses on the .NET CLFS API. You can download the sample application attached to this article to follow along and experiment as you go. The sample application (see ) demonstrates the major topics discussed in the article, such as how to create a CLFS log, establish a log policy, and write to and read from the log.

Creating a CLFS Log
Creating and writing to a CLFS log is similar to creating and writing to a file using the System.IO namespace. For example, to open a file with read and write capabilities, you could write (in C#):

 FileStream stream = new FileStream( "MyApplication.data", FileMode.OpenOrCreate, FileAccess.ReadWrite); 

In the preceding line, MyApplication.data is a file, and the FileStream instance is a “window” into the file used for reading and writing data. You use the same general approach to create a CLFS log:

 LogRecordSequence sequence = new LogRecordSequence( "DevX.CLFS.Log", FileMode.OpenOrCreate, FileAccess.ReadWrite); 

In this case, DevX.CLFS.Log is a log store, and the LogRecordSequence instance is a window into the log store that you use to read and write log records. CLFS automatically appends a .blf extension to the log file. You should always open the log for both reading and writing even if you only intend to write to the log, because CLFS must be able to retrieve information from the log store to operate properly. The LogRecordSequence class implements IDisposable, so be sure to call Dispose() to clean up the resources when you’re done with it.

The first parameter in the LogRecordSequence constructor is the path to the log file. For example, passing in “DevX.CLFS.Log” as shown above stores the log in the same directory as the running application. Passing in a full path stores the log in the specified directory.

Log stores have to start with at least two extents, so add them the first time you create the log.

 if (sequence.LogStore.Extents.Count == 0) { const long EXTENT_SIZE = 512 * 1024; sequence.LogStore.Extents.Add("Extent0", EXTENT_SIZE); sequence.LogStore.Extents.Add("Extent1"); }

If you look at the folder where you created the logs, you can see the new DevX.CLFS.Log.blf file as well as the two new extents (see ).

Before you start writing to the log, you need to establish the log policy.

Establishing a CLFS Log Policy
A CLFS log policy is a collection of settings that determine the characteristics and automatic behavior of your log. The LogPolicy class encapsulates three categories of policy settings: automatic growth, extent management, and tail pinning. You manage automatic growth via log policy properties (see Table 1).

You manage the extents via log policy settings as well (see Table 2).

I’ll cover an additional property, the LogPolicy.TailPinnedThreshold and the LogPolicy.TailPinned event later in this article.

 sequence.LogStore.Policy.AutoGrow = true; sequence.LogStore.Policy.AutoShrinkPercentage = 25; sequence.LogStore.Policy.GrowthRate = new PolicyUnit( 2, PolicyUnitType.Extents); sequence.LogStore.Policy.MaximumExtentCount = 50; sequence.LogStore.Policy.MinimumExtentCount = 2; sequence.LogStore.Policy.NewExtentPrefix = EXTENT_NAME; sequence.LogStore.Policy.NextExtentSuffix = sequence.LogStore.Extents.Count; sequence.LogStore.Policy.Commit(); 

Note the last line in the preceding code. The policy settings don’t take effect until you call the LogPolicy.Commit method.

Now you’re ready to start writing to the log.

Writing to a CLFS Log
CLFS does not define a format for log entries. You’re free to define a format that fits the needs of your application. Fortunately, the .NET serialization API does most of the work for you. Start by defining a serializable class that encapsulates the fields in your log entries. As an example, consider this serializable LogEntry class with four properties (a log entry class for your application might have a completely different set of fields):

 [Serializable] public class LogEntry { private string _subsystem; private int _severity; private string _text; private DateTime _timestamp; public string Subsystem { get { return _subsystem; } set { _subsystem = value; } } public int Severity { get { return _severity; } set { _severity = value; } } public string Text { get { return _text; } set { _text = value; } } public DateTime Timestamp { get { return _timestamp; } set { _timestamp = value; } } }

This example demonstrates creating a log entry that consists of two strings, an integer, and a timestamp, but your log entry class might be considerably more complex. The point to remember is that as long as your class is .NET serializable, it’s compatible with the CLFS.

You serialize an instance of this class to a stream of bytes using a BinaryFormatter and then write those bytes to the log using a LogRecordSequence. The LogEntry.Serializable attribute and the BinaryFormatter class handle all the serialization details:

 LogEntry entry = new LogEntry(); entry.Subsystem = "Transactions"; entry.Severity = 2; entry.Text = "An exception was thrown ..."; entry.Timestamp = DateTime.Now; using (MemoryStream stream = new MemoryStream()) { // Serialize the entry BinaryFormatter formatter = new BinaryFormatter(); formatter.Serialize(stream, entry); stream.Flush(); // Write it to the log ArraySegment bytes = new ArraySegment(stream.GetBuffer()); sequence.Append(bytes, SequenceNumber.Invalid, SequenceNumber.Invalid, RecordAppendOptions.ForceFlush); } 

As shown in the preceding code, the LogRecordSequence.Append method takes four parameters. The first parameter is the serialized log entry. The second and third parameters are used to create relationships between log records (covered later in this article). The fourth parameter controls buffering?RecordAppendOptions.ForceFlush causes CLFS to write the log record to the file immediately. Another option, RecordAppendOptions.None writes the log record to a buffer. Call LogRecordSequence.Flush to write all buffered log records to file. The CLFS guarantees flushed records even in the event of a system failure, so if you’re writing multiple log entries, they aren’t protected from system failures until you flush the buffer.

Reading from a CLFS Log
You use much the same approach to read log entries that you used to write them:

 List entries = new List(); BinaryFormatter formatter = new BinaryFormatter(); IEnumerable records = sequence.ReadLogRecords(sequence.BaseSequenceNumber, LogRecordEnumeratorType.Next); foreach (LogRecord record in records) { LogEntry entry = (LogEntry) formatter.Deserialize(record.Data); entries.Add(entry); } 

Passing the LogRecordSequence.BaseSequenceNumber and LogRecordEnumeratorType.Next parameters to LogRecordSequence.ReadLogRecords() returns log entries in the order in which they’re stored in the extent files.

Creating Log Record Relationships
You could use the Subsystem property on the custom LogEntry class to create relationships between log records. For example, suppose you decide that all the log entries in the “Transactions” subsystem are related to each other. You could find them all by iterating through the log, locating the log records that fit into this category. Unfortunately you would have to iterate through every record in the log, which could be an expensive and lengthy operation for large logs.

A better solution is to create log record relationships. CLFS has a built-in mechanism for creating such relationships between log records, and it’s probably faster than any custom solution. Here’s how it works. Every log record in a CLFS log has a unique sequence number. A LogRecordSequence doesn’t have to iterate through log records to locate one with a given sequence number; it can advance directly to it.

The sequence numbers are similar to pointers or references. CLFS uses the sequence numbers to provide support for two logical record sequences that you’re free to use as you wish: the Previous sequence and the User sequence. These sequences are similar to linked lists.

For example, shows a set of relationally-linked log records. There are two arbitrary sequences in the diagram. The Previous sequence consists of log records 03, 02, and 01. The User sequence consists of log records 03 and 01. You establish record sequences using the LogRecordSequence.Append method. For example, to add a new record, you would write:

 SequenceNumber LogRecordSequence.Append( ArraySegment serializedLogRecord, SequenceNumber userSequenceNumber, SequenceNumber previousSequenceNumber, RecordAppendOptions recordAppendOptions); 

The Append method uses the second parameter to establish User sequences and the third parameter to establish Previous sequences. For example, to build the User and Previous sequences for the three records shown in , you would write:

 SequenceNumber sn01 = sequence.Append(entry01, SequenceNumber.Invalid, SequenceNumber.Invalid, RecordAppendOptions.None); SequenceNumber sn02 = sequence.Append(entry02, SequenceNumber.Invalid, sn01, RecordAppendOptions.None); SequenceNumber sn03 = sequence.Append(entry03, sn01, sn02, RecordAppendOptions.None); sequence.Flush(); 

To read sequences from the log use the LogRecordSequence.ReadLogRecords method, passing in a LogRecordEnumeratorType value to control the read sequence:

 IEnumerable LogRecordSequence.ReadLogRecords( SequenceNumber start, LogRecordEnumeratorType logRecordEnum);

The possible LogRecordEnumeratorType values are:

• LogRecordEnumeratorType.Next?enumerates through log records according to their physical ordering in the extents.
• LogRecordEnumeratorType.Previous?enumerates over the Previous sequence.
• LogRecordEnumeratorType.User?enumerates over the User sequence.

The trick is to locate the correct starting sequence number. There are six places in the CLFS API where you can get a sequence number.

• LogRecordSequence.BaseSequenceNumber is the sequence number of the first physical record in the log.
• LogRecrodSequence.LastSequenceNumber will be the sequence number of the next appended record.
• LogRecordSequence.Append returns the sequence number of the appended record, which you can save for future use.
• LogRecord.SequenceNumber returns the log record’s sequence number. This is the same sequence number returned from LogRecordSequence.Append.
• LogRecord.Previous returns the next log record in the Previous sequence.
• LogRecord.User returns the next log record in the User sequence.

If you don’t wish to specify a sequence number, use SequenceNumber.Invalid. If you don’t specify a sequence number for a log record’s Previous or User sequence, the default value is SequenceNumber.Invalid.

Managing a Growing Log
LogRecordSequence.Append throws a SequenceFullException if your log runs out of space. You can trap this exception and take the appropriate steps to make room in your log or on your hard drive. How you handle this exception is up to you.

• Call LogRecordSequence.AdvanceBaseSequenceNumber to move the base of the log forward and effectively mark older log records as no longer needed. The CLFS will overwrite the log records with older sequence numbers and thus increase the available space in your log.
• Add additional extents.
• Perform other actions appropriate for your application.

For some applications, it’s important to group multiple log records together in a transaction. In this case, you need the CLFS to either write all the related log records or none of them?writing a portion of them would constitute data corruption. The CLFS doesn’t directly support transactions, but you can reserve space for any number of log records before writing to the log.

 ReservationCollection reservedSpace = sequence.CreateReservationCollection(); reservedSpace.Add(sizeOfEntry01); reservedSpace.Add(sizeOfEntry02); reservedSpace.Add(sizeOfEntry03); SequenceNumber sn01 = sequence.Append( entry01, SequenceNumber.Invalid, SequenceNumber.Invalid, RecordAppendOptions.None, reservedSpace); SequenceNumber sn02 = sequence.Append( entry02, SequenceNumber.Invalid, sn02, RecordAppendOptions.None, reservedSpace); sequence.Append( entry03, SequenceNumber.Invalid, sn03, RecordAppendOptions.ForceFlush, reservedSpace); 

The ReservationCollection.Add method throws a SequenceFullException if your log runs out of space. Catching the exception lets you avoid writing fewer than all three related entries to the log.

Sharing a CLFS Log
If you have more than one enterprise class application writing information to a log, maintaining completely separate logs for each application might be more trouble than it’s worth. If the applications are writing a high volume of information to each log, there could be a performance hit associated with the high cost of file operations.

The CLFS addresses this problem with multiplexing, letting two or more applications share the same log file and container files. Multiplexing happens behind the scenes?your applications don’t have to do anything special to share the physical log files with other applications.

Establish a multiplexed log by scoping the log file name with the “::” operator, for example:

 LogRecordSequence sequence01 = new LogRecordSequence( "DevX.CLFS.Log::Sequence01", FileMode.OpenOrCreate, FileAccess.ReadWrite); ... LogRecordSequence sequence02 = new LogRecordSequence( "DevX.CLFS.Log::Sequence02", FileMode.OpenOrCreate, FileAccess.ReadWrite);

The log file of a multiplexed log itself is the same?DevX.CLFS.Log in this case?but has more than one log record sequence written to it. You can read from the log using any of the sequences; the separate log record sequences are independent of each other even though the CLFS persists them in the same files.

Before Microsoft created the CLFS, developers’ logging options included using the Windows Event Log, the Enterprise Library’s Logging Application Block, buying third-party logging framework, or rolling their own solutions. The first two options are often adequate, but lack the robustness and control offered by the CLFS API. In short, Windows Vista and Windows Server 2003 R2 now provide an operating-system based logging solution that allows you to focus on what you want your application to log, rather than on how to accomplish the logging.

About Our Editorial Process