Tips for USB pen drives

Problems with USB drives

There are many possible problems with USB drives. Here are some problems and solutions:

Troubleshooting for USB drives

General

An USB pen drive (also called 'flash drive', 'memory stick' or in german 'USB-Stick') is drive and media in one. Therefore the drive letter comes and goes with it. In contrast a card reader stays attached and only the card is plugged and unplugged, so the drive letters stay.
Because it's drive and media in one, each time you plug a pen drive for the first time the Windows plug'n play starts up a assigns the first available drive letter. When the USB has no USB serial then this happens again if it's attached to a different USB port.
To get rid of the 'first free letter assignment' I wrote the USB Drive Letter Manager which works under Windows 2000, XP and higher.

How Windows assigns drive letters

For a drive which Windows has no drive letter assignment stored it assigns the first available drive letter. The search goes upwards in the alphabet and starts at 'A' for floppy drives, at 'D' for CD-ROM drives and at 'C' for all other types.
Windows can save exactly one assignment per drive letter. So when a drive is attached again it gets the same letter as before. But the drive letter is not reserved while a drive is not present and therefore reassigned to another drive if required. When this happens the former assignment is overwritten and the first drive gets the first available drive letter again even its former drive letter is available when it's being attached. The former assignment just doesn't exist anymore.
Network and substs drives are completely unconsidered by XP here. XP has no problem assigning a drive to letter which is currently used as network share! Microsoft knows that this is 'a problem' and since December 2007 a hotfix is available (WindowsXP-KB297694-x86-ENU.exe). The Service Pack 3 fixed it too (more or less). Not fixed this this scenario:
Local drives C: and D:, USB drive at E:. USB drive is removed, a network share at E: created, the USB drive reattached... XP with SP3 assigns E: again to the USB drive, even E: is used as a network share. This is fixed since Windows Vista.

To get persistent drive letters for external drives assign one exclusive 'high' letter per drive.

My USB Drive Letter Manager solves all this.

Removable or what?

USB drives can appear as 'Removable Drive' or as 'Local Drive'. The actual meaning of 'removable' is that the drive has a removable media, like floppy drives or flash card readers.
Even USB pen drives have no removable media they pretend to have by having the removable media bit (RMB) set in their hardware device descriptor. Strictly spoken this is just wrong but in fact nearly all USB flash drives have the RMB set while USB hard disks have not.
Appearing as removable or local drive makes a difference under Windows. Here the differences for XP up to Windows 10:

Write Cache or not

Under Windows XP writing of lots of small files to a 'Removable' pen drive is extremely slow, while it's fast under Windows 2000. Writing large files there is no appreciable difference.

Obviously Windows XP doesn't enable a write cache for USB drives that appear as 'Removable'. The option "Enable write caching on the disk" is grayed out for USB drive (this mean the hardware cache of the drive) and the removal policy setting ('Quick removal' or 'Optimize for performance') doesn't seem to make any difference, except that the latter enables the user to format 'Removable' USB drives with NTFS. But with NTFS Windows XP enables a write cache, writing small files becomes lightning fast.

Update Feb 2011
I was pointed to the fact that by means of the registry value WriteCacheEnableOverride a write cache can be forced for any USB drive, even FAT formatted removeable drives. And it indeed works!
I have made a litte tool: USB-WriteCache

Update Feb 2011
Since Windows 8 the USB disk device policies have the expected effect on NTFS formatted removable drives, 'Better performance' actually activates a write cache while with the default policy 'Quick removal' there is none (in contrast to XP..Win7). The policies still have no effect on FAT formatted USB removable drives. Here WriteCacheEnableOverwrite is required.

Use NTFS anyway for USB flash drives?
Advantages:

• Write-cache on all XP..Win7 computers or when policy 'Better performance' is set under Win8/10
• NTFS stores small files together with the directory entry, this safes write accesses
• NTFS supports file compression, this safes write accesses too (for compressible files only)
• FAT/FAT32 becomes linear slower with the number of files in folder, NTFS does not
• FAT/FAT32 is limited to 65535 8.3 uppercase named files per folder, non-uppercase and longer file names decrease the number further
• NTFS supports files >4GB

But actually NTFS isn't suitable for flash medias because as journalling file system it has some overhead that wears out flash memory. But Windows XP optimizes and bundles write accesses to pen drives only when they are NTFS formatted, so I see NTFS as the right choice.
Furthermore NTFS stores small files together with the file information so they are written into the same flash block which is the best that can happen.
A drawback with NTFS and flash medias is that NTFS saves the access time when you read a file, so a simple read access causes flash wear out. In fact it is not as dramatically because Windows writes the data not immediately onto the media. It does it when it has to update something else there, when one hour is over on when a media is unmounted. Actually I've never seen XP writing a last access time to a USB drive.
The last access time is also updated on 'directory listing' (whatever this means). This can be disabled, unfortunately for all drives only:
Create a new DWord value under [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem] called NtfsDisableLastAccessUpdate and set it to 1. Or download the following REG file and doubleclick it:
NtfsDisableLastAccessUpdate1.reg
Since Vista this can be configured more detailed: NtfsDisableLastAccessUpdate at Microsoft

To format a pen drive with NTFS there is no need to switch to 'optimize for speed'. Using the commandline tool CONVERT you can convert a FAT formatted drive to NTFS. Sample:

convert u: /fs:ntfs

But XP's CONVERT creates a NTFS with a cluster size of 512 Bytes if the FAT clusters are not 4K aligned. If the drive is FAT formatted under XP, then the clusers are 4K aligned and XP's CONVERT works fine. W2K's CONVERT seems to creates 512 Bytes clusters in any case.
Using the Windows format dialog you can choose: 4096 Bytes is the defaut. With a cluster size above 4096 Bytes file compression and encryption become unavailable.

Of course a NTFS formatted drive doesn't work under Windows 95/98/ME. Furthermore devices with a standalone functionality as MP3 players or image tanks may not work with NTFS.

If both, format and convert fail while formatting with FAT/FAT32 works, then the drive is a fake with a fraction of real memory only. The NTFS file system is written into the middle of the drive while the FAT is written to at the begin of the drive. So, if the drive is (a simple) fake then the NTFS file system is written into non existent memory. Newer fake drives are made more clever, the first write accesses go to existing memory, so formatting with NTFS does not fail. But as soon as more data then really build in is wriiten in sum, the problems begin.

The effect of having a write cache or not depends on the USB drive: Hard drives have an 2..64 MB of cache RAM which prevents to much performance impact. Most USB flash have no such cache, so their access times are very important for their real live speed.
By means of HD Tune you can check the read access times. Here a screenshot of the bad example 'Corsair Flash Voyager' (2GB) - have a look at the yellow dots above the diagram...
It's from 2005, the current exemplars are better.

To test the file cache behaviour I've made FileCacheTest
It can read and write a test file using different flags for the API function CreateFile and using different block sizes. Remember the difference between writing a file for the first time and to an existing file: Writing the first time, the file size has to be adjusted after each block.
The most remarkable effects under XP are:

• no buffering of accesses to the file system on FAT formatted removable USB drives
• writing to FAT formatted removable USB drives using a block size below 4096 Bytes is cached, others are not (for NTFS drives the magic block size is about 512 KB, under Windows 7 32 MB!)
• using flag FILE_FLAG_RANDOM_ACCESS causes a blows up of the file cache's working set
• using flag FILE_FLAG_NO_BUFFERING is the most efficient way to read and write large files but leads to fragmentation if the target file is compressed
• for 'removable' USB drives the removal policies 'Better performance' and 'Quick removal' have absolutely no effect on the cache's behaviour up to Windows 7. Since Windows 8 the settings are effective for NTFS formatted drives but still have no effect on FAT formatted ones.

Deactivate write-cache
There is no known way to deactivate a write cache if Windows decided to activate it, e.g. on NTFS formatted drives there is always a write cache.
Update August 2021: Windows 10 1903 with disk device policy "Quick removal" even NTFS formatted drives have almost no write cache. Only small block writes are buffered (<=32KB per block).

Safe removal

An USB drive should never be removed without logging it off, especially when the drive has a write cache. The official way is thru a symbol in the systray and some mouse clicks.
For 'removable' drives as internal card readers the media should be 'ejected': Right click the drive in the Windows Explorer, select 'Eject' here. Under XP this is not allowed for restricted users but this can be enabled by a policy.

There are some useful commandline tools for this purpose but they all have some disadvantages. So I wrote my own:

• RemoveDrive prepares drives for safe removal
• EjectMedia ejects a media for a drive
• EjectTcv ejects a TrueCrypt/VeraCrypt volume

Reactivate a USB drive

When a drive has been prepared for safe removal then it can be reactivated only by replugging it.
Meanwhile I have learned that this is true only for devices which have the problem code 47 after the safe removal (which is the case after using the safe removal tray icon). Devices which have "Code 21" can be reactivated, I have made a command-line tool for this: RestartSrDev. This works when RemoveDrive was started with admin privileges. Since V2.0 RestartSrDev can restart the USB hub the device is attached to, if required.
Another option is to restart the USB port which is possible under XP, Windows 8 and 10. I have made a command-line tool for this too: RestartUsbPort.
UsbTreeView can do all this too.

My Tools for USB drives

Developing USBDLM I've made some other tools. They got their own page:
Drive Tools for Windows

Data Alignment

SSD and pen drives work with flashmemory which is organized in blocks of a unknown size, probably up to 512 KB, the manufactures don't tell us. For efficient writing the data clusters of the file system should be aligned as good as possible as these flash blocks to avoid unnecessary flash erases.
Therefore new pen drives usually come with a nicely aligned primary partition and a FAT32 file system with aligned clusters, provided the manufacturer made things right...

If a flash drive is partitioned or formatted by the user it is essential to use the right tool. Windows Vista and Windows 7/8/10 do a good job here. They align new partitions and data clusters very well. In contrast Windows XP creates partitions in the classic way at track or cylinder borders which nearly always results in unaligned partitions.

Data clusters: The NTFS file system has its tables somewhere in the middle of the partition, so the data clusters just start at the beginning, no alignment issues here. FAT/FAT32 have their allocation tables at the beginning, the data cluster follow after some 'reserved' sectors. Modifying the reserved sector count is the key to align the data clusters and this is what Vista and Win7/8/10 do very well while XP does a 4K alignment only.
But all this is relative to the partitions!. With non aligned partitions the data clusters are created non aligned too!

I've implemented the display of partition and cluster alignment in UsbDriveInfo. As explained above it shows these information only if they are relevant. For SuperFloppy formatted drives there is no partition alignment and for NTFS drives the first cluster alignment is identical to the partition alignment.

Even with a perfectly aligned first cluster, with a cluster size smaller than the flash block size files are perfectly aligned only with a certain probility: If the cluster have half the size of the flash blocks the every second file is perfectly aligned, with a quarter every fourth and so on. Only with clusters of the size of the flash blocks and at least this alignment full write speed is guaranteed.
But larger clusters lead to more non-usable memory (slack space). And with clusters larger than 4KB there is no more NTFS compression.

By the way: TrueCrypt since V6.0 aligns the data as the container is aligned. From the history: "the start of the data area will always be aligned with the start of a host-filesystem/physical sector"

How to align a partition
Aligning a partition without data loss is reported being possible by means of GParted (based upon a bootable Linux) by moving the partition back and forth, but for a flash drive I think it is not worth the effort. Just delete and recreate it under Vista/Win7/8/10.
On "removable" drives the Windows disk management does no like to delete or create partitions. But the Vista/Win7/8/10 commandline tool DISKPART can.
Another alternative is the SD Memory Card Formatting Software which creates an partition with aligned clusters on USB "removable" drives. The partition is aligned only if the drive is 4GB or larger, so don't format smaller drives with NTFS after formatting them by means of this software!

A nice and free partition software for Windows which supports alignment is the "Active@ Partition Manager". By default it (V2) uses the classic alignment but it offers an option for the exact partition offset. Increase this number to the next value which can be evenly divided by 1024 to get a 512K alignment and decrease the size by the value you have increased the start offset with.
Then it shows a warning plus question "Partition geometry is not aligned with disk track borders. Align on track borders?", we answer "No" because we know aligning to track borders is a thing from the past.
Letting format the drive with FAT resulted here in a 1K cluster alignment. Formatting again with XP (FORMAT or dialog) resulted in the expected 4K alignment. So, better let Windows do the formatting...

What will we get? Depends on the device... With an old 128MB SD card I had no improvement of the write speed at all while a Corsair Voyager 16GB writes large files now with 15 MB/s instead of 6.5 thanks to aligned 64K clusters.

Automatic actions on plug in and out

My USB Drive Letter Manager can execute AutoRun events on arrival of a drive, on removal request and after removal.

Automatic actions on arrival of a USB drive can be easily abused to copy interesting data from it in the background. So, never, never attach a USB drive with sensible data to a foreign computer even if you stand beside all the time! It's very easy to copy all DOC, XLS and PDF files in the background!

USB drive as trojan horse

Have a look at a found USB pen drive - who can resist...
With a normal pen drive there is a danger only under Windows 2000. Since XP the line open= in the autorun.inf file is no more executed by windows. But XP does it on CD-ROM drives! Therefore these U3 flash drives use a fake CD-ROM drive on their pen drives which launches the 'U3 launchpad' automatically. But the contents of the CD-ROM can be easily changed into malware!.
So, whenever you attach a foreign USB drive, hold down the Shift key to skip AutoRun. Or completely deactivate AutoRun by means of my tool AutoRunSettings
Microsoft's TweakUI is buggy here. It completely ignores the default values, so once used, AutoRun for unknown and network drives gets activated!

But there are more dangerous mechanisms, see here:
How to: Quick intro to hacking autorun for USB flash drives

In August 2008 Microsoft finally recognized that there is a security problem and offers updates which apply the autorun settings to the manual actions too, see here:
http://support.microsoft.com/kb/953252

BadUSB

USB flash drives today use quite flexible micro controller which can do way more than acting as a USB mass storage device. For the Phison 2251-03 all tools required to turn it into a BadUSB device have been published:
How it works:
The USB flash drive acts as a USB composite device hosting the expected USB mass storage function but also a bad USB keyboard. Windows comes with standard driver for USB keyboards, so they are installed and get ready without further user requests. Some moments later the keyboard devices "plays" a keyboard macro, ususally pressing Win+R, entering CMD and pressing Enter to open a command prompt. Then they enter a command to eithter start a bad executable from its mass storage device or download a bad executrable from the internet an start it.
Easy counter measures:

• When attaching unknown USB flash drives press and hold the Alt, Shift or Ctrl key, this should screw up the Win+R keystroke.
• Install USBDLM and set BadUsbWatchKbd=1 -> USBDLM asks then on arrival of any USB keyboard, if it shall be activated or not

Other counter measures Windows XP:

• delete or rename usb.inf -> No more new USB hubs and composite devices can be installed
• delete or rename keyboard.inf -> No more new keyboards can be installed

Other counter measures Windows Vista,7,8,10:

• Block new device of certain device classes by registry entries:
  {4D36E96B-E325-11CE-BFC1-08002BE10318} (GUID_DEVCLASS_KEYBOARD)
  {4D36E972-E325-11CE-BFC1-08002BE10318} (GUID_DEVCLASS_NET)
  REG file for keyboard devices
  REG file for net devices
  

Pen drive under Windows 98 and NT4

For MS-DOS there is a lot of USB stuff. This page gives a good summary:
USB DOS Ressources
This one too:
USB Bootdisk

For Windows 98 you get drivers with the drive. If not you can try to use a driver of a foreign drive. Here is a german howto:
http://www.techwriter.de/beispiel/usb-mem2.htm
From COS Memory there are drivers which can be adopted in the same way:
http://www.cos-memory.de/downloads/treiberdownload.htm

Meanwhile there is an universal driver package for USB drives:
http://www.technical-assistance.co.uk/kb/usbmsd98.php
It uses Windows 2000 files, so you should have an additional valid Windows 2000 licence when you use this package...

Windows NT4 got no USB support by Microsoft - we shall buy a new system from time to time... Other vendors help:

• USB driver by Inside Out Networks for Dell Latitude Laptop (doesn't work with USB hubs. Using an additional package by Inside Out Networks you can add hub support: ip4n4260.exe
  There seems to be no read or write cache with NT4, so pen drives behave very slow here.
  
• From Digi International there is a driver package for a USB camera which shall include a driver for USB mass storage. I havn't tested it.
  http://ftp1.digi.com/support/driver/i4usb406.exe With files from i410usb.rar this can be updated to V4.10.
• From China comes another USB driver for NT4 which supports USB 2.0 highspeed. I've made a short test: On a NForce2 board it hangs sometimes but I think that's a NForce2 problem. On several other machines it worked fine and fast. Read and write cache is active. The only drawback is that it assigns always drive letter Z: even according to the homepage this should be fixed. You can change this with the disk manager but on next reboot it's again Z:. Woodhead's Home
• One more page: Windows NT 4.0 and USB

Encrypt a USB drive

All encryption solutions which work thru an encrypted drive with drive letter have on thing in common: Administrator privileges are required, at least once to install the ecryption software. This is a problem on foreign computers.
When a guest wants to install his encryption software, he must been trusted unconditional because he is granted to execute his software with administrator privileges. But no one can be sure what this software does despite the encrytion job. The computer is just compromised then!
Even an antivirus software is useless because the foreign software is executed with admin privileges an the user will grant all requests to make it work.

NTFS encryption is no help too because dealing with certificates is no fun and they are not compatible between Windows 2000 and XP.

Solution: The approved open source software TrueCrypt V7.1a: https://www.grc.com/misc/truecrypt/truecrypt.htm
or
https://truecrypt.ch
It requires admin privileges too but only once for installation.

In May 2014 the TrueCrypt developers lost interest in developing the software and declared it as potential unsafe because "it may contain unfixed security issues". An audit of the TrueCrypt source code didn't reveal any real issues, so V7.1 can be assumed as safe.

Using TrueCrypt is documented on countless pages in the internet, just enter TrueCrypt at your preferred search engine.

Meanwhile there is a new project based upon TrueCrypt: VeraCrypt

Links

HP USB Disk Storage Format Tool

SD Memory Card Formatting Software

USB flash drive FAQ

Booting from USB

Fixing Windows 2000/XP Drive Letters

Letter Assigner for Windows 95/98/ME