The XDF Diskette Format

In 1994, IBM started shipping software with support for XDF, or eXtended Density Format, first in OS/2 Warp and a few months later in PC DOS 7.0. Some of IBM’s software packages were also distributed on XDF diskettes. XDF allowed the user to format a 31⁄2” floppy which normally holds 1,440KB of data to a special format with 1,840KB capacity (well, almost—see below), or almost 28% improvement. This came at a time when CD-ROMs were not yet ubiquitous and software was distributed on rapidly growing piles of floppies. Since floppies were relatively expensive to manufacture and duplicate, shipping software on, say, 16 diskettes instead of 20 was an attractive proposition.

The XDF technology was not developed by IBM. The inventor was Roger D. Ivey (as can be seen in a XDF disk’s boot sector) and was first licensed to IBM by Backup Technologies, Inc. of Tampa, FL. Roger Ivey had previously written the Fastback PC backup software. Later releases of XDF software list Ametron Technologies, Inc. as the licensor. Ametron is a company specializing in selling intellectual property.

The XDF technology used several interesting techniques to achieve both higher storage capacity and speed. The XDF utilities mentioned “Patent(s) Pending” but there is no obvious record of a relevant patent ever being granted by the USPTO. Let’s take a look at some of XDF’s tricks.

The basic problem with higher density floppies when using 512-byte sectors is that once more than 10 or so sectors per track are used, significant capacity is lost to inter-sector gaps, address marks, CRCs, an other “useless” overhead. With 31⁄2” HD floppies, it is possible to use more than the standard 18 sectors per track; the hard limit appears to be 21 sectors which translates to 1,680KB total with 80 tracks. XDF squeezes another 160KB into the same space, using standard hardware and media.

Note that the rest of the article talks about the 1,840KB XDF format unless otherwise noted. XDF also supported a 1,520KB format for 51⁄4” HD floppies (normally 1.2MB) and a 3,680KB format for 31⁄2” ED media (normally 2.88MB).

XDF Format

The basic XDF approach involved the use of larger sectors, which reduced the sector overhead. On a 31⁄2” HD floppy (DD media were not supported by XDF), where the standard format used 18 sectors per track (9KB), a XDF track contained the equivalent of 23 sectors (11.5KB). Physically, the track was formatted with only four sectors in 8KB, 2KB, 1KB, and 512B sizes (for the 11.5KB total). With an unformatted capacity of about 12,500 bytes, there was just not quite enough space for one 8KB and one 4KB sector (let alone a single 16K sector), while the 8/2/1/0.5 breakdown was reliable.

From reading typical floppy controller documentation, it is not obvious whether a standard FDC would be able to create XDF diskettes. The FORMAT TRACK command takes the sector size as a parameter for the entire track, yet the data supplied to the command for each sector also includes the sector size code. Does the FDC observe the per-track sector size or the per-sector one? FDC documentation is typically annoyingly quiet on this.

The Hitachi HD63265 FDC user’s manual is one of the rare documents which addresses this. Hitachi unambiguously says (on page 56 of the manual) that the actual sector size will be as specified for the entire track, but the size code written to the sector ID fields will be as supplied for each sector. In other words, the sectors will always be the same size on a track. If that’s true, how did XDF format tracks with non-uniform sector sizes?

XDF—of course—cheated. When formatting a XDF track, the FDC was told to create 39 128-byte sectors. However, the sector sizes written to the sector IDs were entirely different. When the sector would be later written, the FDC would only pay attention to the sector size code in the ID field (as there’s no other sector size information on the medium) and happily overwrite the gaps, sector IDs, sync fields, and CRCs of the “fake” 128-byte sectors.

This way XDF circumvented the limited FDC capabilities and created a very unusual but valid format with standard floppy hardware.

The special sector layout was key for achieving high storage capacity. In addition to varying sector sizes, XDF used interleaving and sector slipping to speed up XDF media reading and writing. As a result, XDF was faster than alternative high-capacity formats.

DOS Implementation and Compatibility

On DOS, a XDF.COM utility (TSR) provided standard DOS access for XDF-formatted floppies. The XDFCOPY.EXE tool was used to copy XDF disks, create images from disks and write images to disks. As a side functionality , XDFCOPY also handled most standard DOS formats.

Given XDF’s highly non-standard format, one might think that the DOS-based XDF utilities needed to use direct FDC access. They did not—instead, they showed just how far the standard floppy BIOS services can be convinced to go.

For formatting, the BIOS user supplies a buffer containing per-sector parameters that the FDC then transfers during execution of the FORMAT TRACK command. For reads and writes, XDF dynamically changed the DPT (Diskette Parameter Table) and used the standard BIOS read/write services to access one sector at a time.

It is also apparent that without the XDF TSR loaded, BIOS/DOS won’t be able to do much with XDF media. To this end, XDF implemented a clever backwards compatibility scheme.

When the XDF TSR was loaded, a 31⁄2” HD XDF floppy looked like a medium with 80 cylinders and 23 sectors per track. When the XDF TSR wasn’t loaded, the same medium looked like a teensy disk with just a few sectors total, usually containing some text file showing the XDF disk’s contents or a readme file.

The first cylinder of a XDF floppy used standard 512-byte sectors, only with 19 sectors per track rather than the standard 18. The format used 11 sectors per FAT (for the 31⁄2” HD XDF variant, by far the one most widely used). It abused the fact that the second FAT is normally never read by DOS (as long as the first FAT copy is readable). The second FAT copy on a XDF disk therefore did not exist at all, but instead there was a small 8-sector backwards compatible image. The XDF sectors used high sector IDs and were thus not seen by standard DOS/BIOS. Track 0 (cylinder 0, head 0) was laid out as follows: 1 boot sector, 10 FAT sectors, 8 sectors of a backward compatible micro-disk. That is 19 sectors total.

The second track (cylinder 0, head 1) started with the one last FAT sector (for a total of 11), followed by the root directory (14 sectors) and finally the data area (4 sectors). Now there’s an obvious problem: The XDF disk claimed to have 23 sectors per track, therefore there should have been 9 data sectors (23 – 14) on the second track, but there were only 4. What happened with the missing 5 sectors? Easy: XDF marked the corresponding clusters in the FAT as bad.

To recap: XDF pretended that the first track (cylinder 0, head 0) contained the boot sector plus two FATs with 11 sector each (23 total). The next track (cylinder 0, head 1) then logically contained 14 sectors of root directory and 9 data sectors (23 total), out of which 5 were marked bad. In reality, the first track held 8 sectors of a compatible micro-disk and 11 sectors of XDF data (19 total); those were the XDF boot sector and the first 10 sectors of one FAT. The second track actually contained the last sector of FAT, 14 sectors of root directory, and 4 data sectors (19 total). Starting with cylinder 1, there was really the equivalent of 23 sectors (512 byte size) per track.

It is important to note that XDF images always contained full XDF-sized tracks, even for cylinder 0. Some sectors in a XDF image were thus not written to a disk, but the XDF image was uniform.

One pleasant side effect of the XDF scheme is that a typical XDF image looks like a proper DOS-formatted disk with 23 sectors per track and 3,680 512-byte sectors. Since the FAT filesystem uses logical sector numbers rather than C/H/S addressing, tools capable of processing FAT disk images can use XDF images without modification.

A further side effect is that when XDF images are used with virtualization software, DOS VMs may be able to access such images without the XDF TSR loaded.

Alternatives

Around the same time, Microsoft developed its own high-capacity format called DMF (Distribution Media Format). DMF was still a standard format on the BIOS level, with 80 cylinders and 21 sectors (interleaved) per track. Since DMF was optimized for distribution, it used larger clusters (and hence smaller FATs) and a minimal root directory with only 16 entries (one sector). This made DMF unsuitable as a general-purpose floppy format. At 1,680KB per disk, DMF also had noticeably lower storage capacity than XDF.

XDF’s closest cousin was the 2M format used by the eponymous DOS utility. The 2M format could store up to 1,972KB (or just over 2 million bytes) on a 31⁄2” HD floppy. 2M used a very similar strategy utilizing varying size sectors and thus minimizing sector overhead. Additionally, 2M used 82 tracks rather than the standard 80. It is questionable whether the use of 82 tracks was fully compatible with all drives. At any rate, the 2M format does not appear to have been used for commercial software distribution.

It is not entirely clear whether 2M or XDF was developed first, and whether their development was independent. Both were most likely initially written in 1993. It is apparent that XDF was considerably more successful, having been adopted by IBM for use in DOS and OS/2 and for distributing IBM software.

This entry was posted in Floppies, IBM. Bookmark the permalink.

24 Responses to The XDF Diskette Format

  1. Darkstar says:

    Very interesting article. I didn’t really know how XDF worked internally.

    Have you ever heard about a backup product called Central Point Backup? It was pretty popular around here back in the nineties, and it also used a special disk format which probably worked similar to XDF. It was also a lot faster than regular DOS floppies (same interleaving?) and contained a basic backwards-compatible root directory which contained a few 0-byte file entries with file names that, on a standard “dir”, spelled out that the disk is a CPBackup disk and can only be used with that program.

    It would be interesting to compare that format to XDF

    –Darkstar

  2. Michal Necasek says:

    Not only have I heard about CP Backup, I actually used it 20 years ago. I recall that it had several modes of operation and that the “native” mode was extremely fast. I only ever used CP Backup with 5.25″ floppies but I clearly remember that it could read the entire disk 2x-3x faster than DOS. Sector slipping and possibly interleaving is how I suspect CP Backup got its speed.

    I don’t think the “special-format” floppies had significantly higher capacity, so I suspect the format wasn’t quite as interesting as XDF. But I’ll try to find my old backup floppies and run them through AnaDisk.

  3. Rauli says:

    As I knew the author of 2M, I can assure that development of 2M was independent of XDF. He used his experience on PC hardware and the ideas of an electronics teacher in the university where we studied.

    Also I remember that at some point of 2M evolution (possibly between versions 2 and 3) he started to receive e-mails from IBM. Were they interested on getting 2M (or him)? I don’t know. I know that the final messages were about “after many hours continuously formatting disks with the 2M formatter the drive starts to melt the disks”.

    He said that “they look very interested in me saying that 2M formatter is the cause for the disk melting, and I’m not going to say it, so I have stopped replying them”. He did not try to verify what IBM claimed, but he said that after many hours of formatting, if the drive starts to melt disks, it must be the drive’s problem and it would melt disks regardless of the formatting sofware used.

    If you are interested I can look the dates of 2M source code or more information in the home-made books of the author (in the chapter about the diskette controller he explains how 2M works).

  4. Rauli says:

    About the use of 82 tracks: At the beginning of the 2M era, as it had a command line option to specify the number of tracks, I formatted all my diskettes using 84 tracks. Unfortunately, when I changed computer, I noticed that the new drive couldn’t access the last 2. So I started to try any drive on any PC I could use. I only found another drive supporting 84 tracks, but it seemed that every drive those days supported 82 tracks. But I stopped testing drives and using 2M when Windows 95 arrived (as it always re-writes the boot sector, it destroys 2M disks).

  5. Michal Necasek says:

    It would be interesting to know if 2M used the same method of formatting tracks with varying sector sizes, and when exactly it was developed. Pinning down the XDF origins might be a lot harder though.

    The melting disk stuff sounds like an urban legend… but then again I’ve seen a CD explode in a CD-ROM drive, so who knows.

  6. Michal Necasek says:

    The problem is that even if just one drive out of 1,000 can’t do 82 tracks, that’s enough to kill the idea of using 82-track distribution media. Given that neither IBM nor Microsoft used it, I suspect some drives did not do 80+ tracks. Then again it’s possible that the companies were simply conservative because they couldn’t verify that every drive out there could handle 82 tracks.

  7. Rauli says:

    The last printed edition (3rd) of the book is from 1994 February, and it explains how the last version of 2M works. But I must have also the 2nd edition (from 1993 or earlier) and I think it explained a previous 2M version. If I find it, I will say the date.

    There is a 1997 edition (4th) in PDF but I think it is a re-formated for PDF version of the 3rd edition (no extra contents). It is free, but it is in Spanish:

    ftp://ftp.cs.buap.mx/notas/programacion/ensamblador/El.Universo.Digital.del.IBM.PC.AT.y.PS2.(4%C2%AA.Edicion).pdf

    2M explanation starts in page 309.

    It used “normal” and “maximum” (/F) formats. Normal formats used only 1 Kb sectors. Maximum formats used 2 Kb sectors and 1 Kb and 512 byte sectors to complete the tracks (built on 128 byte sectors, like XDF). Track 0 was “more or less normal” using 512 byte sectors. The resident code reported 2 FAT copies, but the disk only contained one (again like XDF). It ignored writes to the 2nd copy and returned the 1st one when asked to read the 2nd. But track 0 included the resident code and installed it at boot, so you could make a 2M disk bootable.

    And… I have seen melted 5,25″ diskettes, but that was a different story: That was how my uncle noticed that the fan of the power supply was not working. Ant that was the end of the old IBM XT because for the price of a new power supply he could buy a 386 clone.

  8. Richard Wells says:

    The documentation that came with the last DOS release of 2M compares the variable sector size methods of XDF versus 2M. It points to IBM’s method being better able to handle slight differences in the rotational rate of drives while 2M’s method uses less memory. See 2M-INFO.FDA close to the end of either the Spanish or English sections. It also covers some downsides with the various methods.

    The extra tracks was available on drives but some of the information I have seen suggests that the extra tracks were not always long-term reliable when written by the typical floppy drive which makes backing up diskettes problematic. Verbatim’s preformatted floppies put manufacturing information in the extra tracks and caused an amusing bug with Winimage if the diskette had been reformatted but the extra tracks left alone. Toshiba went one step further and supported 84 tracks. The drives that would not handle extra tracks (Flopticals, USB floppies) also do not work with XDF. http://toastytech.com/files/nformat.txt includes one of the few lists of overformatted support by brand.

    DOS backup programs did lots of unusual diskette formats. http://www.oldskool.org/guides/dosbackupshootout dicusses some examples. Magazines during the 80s and 90s had occasional articles investigating how to shoe horn more data on a floppy but I can’t track down references to things I remember reading.

  9. Michal Necasek says:

    All standard USB floppy devices use a SCSI style interface, which definitely does not support any “funny stuff”. Just uniform geometries with constant sector sizes (not always 512-byte sectors though). So yeah, no XDF or 2M or anything like that for sure.

  10. Rauli says:

    BTW, about using different sector sizes on the same track, page 310 says that in theory that’s not possible, but that certain commercial software had used this technique for copy protection. So, that’s probably where he got the “inspiration”.

    I would like to think that when Ciriaco started to be e-mailed from IBM, they were deciding which format (XDF or 2M) to use for software distribution. If the story about melted disks was true (the message was, because I read an excerpt) then one thing is sure: IBM thoroughly tested 2M.

    Tell me if you have something to ask him, we haven’t met for more than 20 years, but I have located his Facebook profile and maybe he will reply.

  11. Rauli says:

    …LESS than 20 years, bad calculation 🙂

  12. Michal Necasek says:

    The fact that commercial software used non-uniform sector sizes for copy protection doesn’t necessarily imply such disks were formatted using a standard FDC…

    If you can, ask the 2M author how exactly he got the not-really-128-byte-sectors formatting idea, because that’s something which is far from obvious. And please ask whether to his knowledge XDF or 2M was the first to use these techniques.

  13. Rauli says:

    I asked him and gave my e-mail address this article’s URL, but his latest news in Facebook is from 2011…

  14. Yuhong Bao says:

    I wonder what would have happened if MS took the OS/2 2.0 path, and thus OS/2 2.x replaced DOS/Windows by 1994 or so. MS would probably have to ship with applications that use DMF an extra disk with the update to OS/2 2.1 to recognize DMF.

  15. Yuhong Bao says:

    Not that it is much of an argument not to take this path.

  16. Jorpho says:

    As you might expect, there’s at least on Linux utility capable of reading XDF disks. I made use of it a few years ago, when a copy of Warp 3 passed through my hands. Oddly, while one of the disks was not entirely readable with XDFCOPY in DOS, the Linux utility seemed to be more tolerant.

  17. Richard Wells says:

    Yuhong: I am fairly certain that I installed software placed on DMF disks on OS/2 Warp with no problems. A DMF driver is tiny. So DMF would not matter in any joint IBM/MS project except for being one more trigger for yet another bureaucratic fight as various managers push their solutions.

    Going the other way, having XDF disks for application deployment would be more problematic. Disk caching software often assumed that all sectors on a disk would be the same size. Caching software would have to be updated in order to permit installs from XDF floppies or users would need to follow instructions to turn off the cache. Special instructions leads to lots of support calls.

    My quick check suggests that IBM only used XDF with OS related disks (Install and fixpack) but not on application software like Lotus SmartSuite.

  18. Michal Necasek says:

    Unless it was on the same machine, it may have been the drive and not the software that made the difference…

  19. Michal Necasek says:

    On the BIOS level, XDF disks (with the XDF TSR loaded) did look like regular 512-sector disks, just with 23 sectors per track. How else would they work with DOS?

    For Lotus et al, a much bigger problem was bootstrapping. Not all systems they targeted supported XDF, which means they’d have to have some fallback mechanism to get XDF support in place. This wasn’t an issue for IBM’s DOS and OS/2 where the installation disks had to be booted anyway.

    And by the time there might be any kind of significant number of machines with built-in XDF support out there, floppies as a distribution medium were dead anyway.

  20. MiaM says:

    In theory, a floppy drive using SCSI, IDE/PATA or USB could support this format if the firmware manufacturer did take the extra effort it takes.

    There are probably no such drives, but we shouldn’t rule it out.

    In some cases it might even be possible to hack the firmware. Atleast Digitals scsi-floppy-interface (used in their desktop VAX computers e.t.c. in the 80’s) has IIRC a standard microprocessor and separate eprom 🙂

  21. Michal Necasek says:

    How do you think it could be done for e.g. USB floppies? Making it look like a disk with lots of 512-byte sectors? That should be technically doable, but I have no doubt that no one ever did it 🙂

  22. Yuhong Bao says:

    That is because OS/2 Warp was updated to support XDF and DMF.

  23. Yuhong Bao says:

    Thinking about it, installing any driver on OS/2 including a updated floppy driver would likely have required a reboot before the DMF floppies can be read.

  24. Yuhong Bao says:

    This states clearly that NT 3.1 can’t read DMF disks and a new floppy driver is required:
    http://web.archive.org/web/20090420082156/http://support.microsoft.com/kb/124970
    Imagine if this new floppy driver had to be shipped with every copy of Office or other DMF applications.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.